0

I was studying Shannon's Entropy function and for a 35% chance of a particular event, the formula produced the answer 1.5 bits.

log2(0.35) = 1.5 bits(approx.) of information.

I know it's pretty trivial in context of practical applications, but how can one visualize 0.5 a bit of information. I mean a bit could be one or zero but 0.5? Analog values ?!?!?

Zaid Khan
  • 111
  • You are supposed to round up. It just means that $1$ bit isn't enough. – Rushabh Mehta Jan 29 '20 at 06:37
  • 5
    You seem to be confusing the set of possible values of a bit (as information stored in some device) and the use of bit as a unit of information. A single coin flip contains one bit of information because the two possible outcomes are equally likely. But if the coin were heavily loaded, say with 99 per cent chance of getting tails, then the outcome of flipping such a coin surely contains less information than that of flipping a fair coin. – Jyrki Lahtonen Jan 29 '20 at 06:40
  • @nicomezi Yeah. That's one of thinking of it. I have always thought the same but tbh in isolation I often think there could be something more in it. I guess that's not true then. – Zaid Khan Jan 29 '20 at 06:41
  • I deleted my comment because I think you should focus on @JyrkiLahtonen explanation. – nicomezi Jan 29 '20 at 06:41
  • @JyrkiLahtonen Thank you. That's nice subtle explanation. – Zaid Khan Jan 29 '20 at 06:43
  • Mind you, in telecommunication we also have so called soft decision bits. When transmitting bits via radio waves, we may assign a certain transmitted signal to mean "1" and another to mean "0". But due to thermal noise and such the receiver is likely to see a distorted waveform that is not quite either of the originals. It then makes sense (in some error-correction schemes) to treat the received "bit" as $0.9$ if it is that much more likely to be $1$ rather than $0$. That is something different though (but still related). – Jyrki Lahtonen Jan 29 '20 at 06:50
  • @JyrkiLahtonen I was actually learning it for understanding how Cross-Entropy is used as a cost function in deep learning. Basically the whole idea of deciding the difference between a true probability distribution and using actual distribution by finding the difference in the bits used vs. bits which actually were needed to carry information. You can watch it here – Zaid Khan Jan 29 '20 at 14:33

2 Answers2

2

Thank you to Jyrki Lahtonen in the comments for answering it convincingly.

You seem to be confusing the set of possible values of a bit (as information stored in some device) and the use of bit as a unit of information. A single coin flip contains one bit of information because the two possible outcomes are equally likely. But if the coin were heavily loaded, say with 99 per cent chance of getting tails, then the outcome of flipping such a coin surely contains less information than that of flipping a fair coin.

Zaid Khan
  • 111
  • 1
    A good alternative if the person is reluctant to turn the comment into an answer is to make your answer a CW answer ("wiki post", or formally "ommunity wiki" post). That way no one gets points when it is upvoted, but the question leaves the unanswered queue (and potentially benefits future users of the site). – almagest Jan 29 '20 at 08:18
  • @MorganRodgers I do not want your upvote. I simply wanted an answer to my question which I find it to be convincing. Also, I don't wanna waste anyone's time trying to answer again because I have already have received a good answer. I also have given him complete credit. – Zaid Khan Jan 29 '20 at 12:36
  • @almagest Nice. I have changed it into a community wiki. – Zaid Khan Jan 29 '20 at 12:37
  • Thanks, and I have upvoted it! – almagest Jan 29 '20 at 14:14
-1

One bit of perfect information will allow a person to double their money. The simplest idea, is if you had a tipster than knew who would win the football game, you could go to a bookie, place a wager and double your money.

Suppose you have a tipster that is right 55% of the time. It is still worth following your tipster's advice, but you are not going to reliably double your money. And, you are not going to wager all of your money on one bet. You have a partial bit of information.

Doug M
  • 57,877