0

If I choose a random alphabet from (a-z), $26$ characters, what is the entropy?

Shannon's formula:

$$H = - \sum p \log_2(p) = - (1/26)\log_2(1/26) = 0.18$$ bits.

However, other formulas on the Internet use:

$$H = \log_2 (N^L) = \log_2(26) = 4.7$$

bits.

Which one is correct?

Hendrix
  • 1,426
Bhagwad Jal Park
  • 113
  • You only got one term of the sum, not all $26$ of them – Ross Millikan Aug 09 '19 at 16:35
  • @RossMillikan Thanks for the clarification! I had wrongly thought that the sum was for the number of characters in the string. Turns out it's for the number of possible characters in the character set :) – Bhagwad Jal Park Aug 09 '19 at 17:17

1 Answers1

2

The internet is correct.

The sum is over all possible outcomes, and not just one. So you should get:

$H = \sum_{x = 'a'}^{'z'} - p(x) \log_2(p(x)) = \sum_{x = 'a'}^{'z'} - \frac{1}{26} \log_2(\frac{1}{26} = 26 * (-\frac{1}{26} \log_2(\frac{1}{26})) = -\log_2(\frac{1}{26}) = \log_2(26) = 4.70043971814...$

btilly
  • 1,186
  • 1
    Put another way, Shannon's formula $H = - \sum p \log_2 p$ is correct as written, it's just that the expansion in OP's question of that sum is incorrect. – Travis Willse Aug 09 '19 at 16:11
  • Understood thanks! I mistakenly thought that the sum was over the number of characters in the string, not the character set itself. – Bhagwad Jal Park Aug 09 '19 at 17:16