0

A previous post has shown that for random variables $X$ and $Y=bX$, where $b > 0$, the entropy of $X$ and $Y$ are not equal (Entropy of $Y=bX$). However, wouldn't any bijection $g$ on a random variable $X$ yield $H(X)=H(g(X))$? It seems logical not to decrease entropy in this case, since we are essentially relabelling the outcomes of $X$.

Community
  • 1
user166224
  • 1
  • Thank you for your help, b yen. I am still working on the details of finite information theory. I mistakenly assumed the problem was dealing with the finite case. It is nice to know that things get stranger and more interesting later on. – user166224 May 17 '15 at 02:28

1 Answers1

1

The post you are referring to is about differential entropy $h(X)$.

It's true that simply relabeling the outcomes of a discrete random variable $X$ cannot change its entropy $H(X) = -\sum_x p(x)\log p(x)$. Differential entropy is a similar quantity, but lacks some of the nice properties of $H(X)$, e.g., it can be negative, it changes with nonzero scaling, etc.

The relation between these two concepts of entropy is discussed at http://en.wikipedia.org/wiki/Differential_entropy

b yen
  • 276