1

Entropy supposedly " is the average amount of information contained in each message received"(Wikipedia: Entropy). However, to calculate the Shannon entropy for a finite sample, we have the sum over the of -p_i(log p_i). However, for each i, the amount of information possible is a positive number of bits, so for one instance one must round down to the nearest integer. Thus I would expect the total sum to be composed of integers; i.e., each term in the sum would be FL(-p_i(log p_i)), where FL is the floor function. Why isn't it?

A similar question then comes up when one considers the properties of the information function I so that I(p)I(q)= I(pq).

Arthur
  • 199,419
nomadreid
  • 352
  • The Shannon entropy of a process/random variable $X$ is the average number of bits required to represent $X$. This is the content of the source coding principle. An average need not be a natural number. – Isomorphism Feb 22 '15 at 13:05
  • @Isomorphism: I don't mean to round off the entire sum nor the parts that calculate the average, but rather to each contribution of bits, which should be integers. If one sample has a probability of p_i = 3/4, then you are not going to get part of a bit of information from it, are you? Perhaps I should have asked whether each term shouldn't be (-p_i(FL(log p_i))). – nomadreid Feb 23 '15 at 13:45
  • Measuring one sample with p=3/4 won't give you 0.811 bits, but measuring 1000 independent such samples will give you about 811 bits. – Oscar Cunningham Mar 22 '15 at 14:32

0 Answers0