0

I was reading this explanation on how to calculate the entropy of a password. The article is great and it explains it very succinctly that even I understood it.

According to the site, if you have a password that has only lower-case characters, you will have a pool of 26 possible characters from the English alphabet. Paraphrasing it further:

Entropy is calculated by using the formula $\log_2 x$, where $x$ is the pool of characters used in the password. So a password using lowercase characters would be represented as $\log_2 26 \approx 4.7$ bits of entropy per character.

If I remember correctly, this logarithmic expression can be algebraically expressed as $2^x=26$: $x$ being the ‘4.7 bits of entropy per character’. Why? What is the property that makes the value to which the base is powered to be the entropy of a character?

vonbrand
  • 27,812
Lex
  • 103
  • In binary, you only have two bits; thus encoding a character which can take 26 values requires $\lceil \log_2 26\rceil$ bits, since the number of possible different encodings using $b$ bits is $2^b$, and you need this to be at least $26$ to cover all possible distinct $26$ characters. – Clement C. Feb 12 '14 at 19:53
  • @ClementC. - In other words, if it were 2 bits instead of 4.7, I would get only four different binary combinations, not nearly enough to cover all 26 characters. Likewise, 3 bits are not enough either (8 combinations), but 4.7 seems just enough to accommodate all 26 possible characters, hence the logarithmic expression? – Lex Feb 12 '14 at 20:05
  • Roughly, yes: this explains the log in terms of covering the space of possible combinations of characters. – Clement C. Feb 12 '14 at 20:43
  • @ClementC. - Thanks so much; you managed to explain something in less than fifty words that I just could not have understood on my own. If you would like to paste your comment as an answer, I will happily upvote it and accept it. – Lex Feb 13 '14 at 11:52

2 Answers2

1

From the comments above:

In binary, you only have two bits; thus encoding a character which can take 26 values requires $\lceil\log_2 26\rceil$ bits, since the number of possible different elements encoded using $b$ bits is $2^b$, and you need this to be at least $26$ to cover all possible distinct 26 characters. This explains why the $\log_2$ shows up here: it is to cover the space of all possible elements (if the alphabet had $a$ different characters instead of the $2$ bits of binary, it'd be $\log_a 26$).

Clement C.
  • 67,323
1

The "entropy" is a measure of the amount of information in a message (the name comes from many of the formulas having an uncanny resemblance to the ones in statistical mechanics for entropy). And customarily "information contents" is measured in bits (2 options), thus the "power of 2" and "logarithm of base 2" which show up all over the place. If martians measure information in trits (3 options), they'll have formulae with 3 all over.

vonbrand
  • 27,812