Nowadays I refer to some references about entropy.They all say "The entropy function $H(X)$ is a concave function".The definition is as follows: Let $X$ be a continuous random variable with probability density function (pdf) $f (x)$ (in short $X ∼ f (x)$). The entropy of $X$ is defined as $$h(X) = −\int f (x) \log f (x) \mathrm{d}x = −E_X(\log f (X)).$$ My question is that the log function is concave , so minus of the log function is convex. So why is the entropy of $X$ concave?
Asked
Active
Viewed 664 times
0
-
Hint: construed as a function of $f$, $-\ln f$ may be convex, but what about $-f\ln f$? – J.G. Aug 04 '22 at 11:49
-
Could you tell me the complete proof for the convexity in detail?Thanks – solver Aug 04 '22 at 13:21
-
Just compute the second derivative with respect to $f$. – J.G. Aug 04 '22 at 13:35
-
So what space does the operation take place?What's the definition of this kind of derivative? – solver Aug 04 '22 at 13:58
-
Could you calculate it rigorously? – solver Aug 04 '22 at 14:30
1 Answers
1
Consider the simple case of the binary entropy $$ H(X) = \operatorname H_\text{b}(p) = -p \log_2 (p) - (1 - p) \log_2 (1 - p)$$ $$\frac{ d^2 H_\text{b}(p)}{dp^2}=-\frac{1}{(1-p)\, p\, \log (2)} <0$$
Claude Leibovici
- 260,315
-
-
@solver, a more general approach to this (i.e. considering the continuous case) requires knowledge of functional derivaties. – venrey Aug 05 '22 at 04:33