1

Being $h$ the differential entropy, defined by $h(X) = - \int f(x) \log(f(x)) dx$, where $X$ is a random variable.

I know there's a property of $h$ that states that: $h(Y) = h(X) + \log|a|$, being $Y=aX$, where $a \ne 0$ is a deterministic constant and $X$ is a random variable.

I was wondering which is the justification of this for the case of:

$Y=aX$, and the PDF of Y $f(y)=(1/a) \cdot f(x/a)$.

ViktorStein
  • 4,838
Llacer
  • 21
  • 2

1 Answers1

2

As you mention, if $X$ has the pdf $f(x)$, $Y$ has the pdf $a^{-1} f(a^{-1}x)$. This can be seen by taking the derivative of the cdf $F_Y(y) = P(Y \le y) = P(X \le a^{-1} y)$, as $\frac{d}{d y} F_X(a^{-1} y) = a^{-1} F'_X(a^{-1} y)$ holds by the chain rule.

Therefore for $a > 0$, \begin{align} h(Y) & = -\int a^{-1} f(a^{-1}x) \log\big(a^{-1} f(a^{-1}x)\big) dx \\ & = -\int a^{-1} f(a^{-1}x) \log\big(f(a^{-1}x)\big) dx - \int a^{-1} f(a^{-1}x) \log(a^{-1}) dx \tag{1}\\ & = -\int f(x) \log(f(x)) dx + \log(a) \cdot \int a^{-1} f(a^{-1}x) dx \tag{2} \\ & = -\int f(x) \log(f(x)) dx + \log(a) = h(X) + \log|a| \tag{3} \end{align} holds.

We use the following three facts:

  1. $\log(a b) = \log(a) + \log(b)$ and $\log(a^{-1}) = - \log(a)$ for $a,b > 0$.
  2. $\int g(a^{-1} x) dx = a \int g(x) dx$.
  3. $\int f(x) dx = 1$, when $f$ is a pdf.
ViktorStein
  • 4,838
  • Hi, thank you for your reply. But why are we taking the value of a >0 and why do you have log of absolute value there? – Karina Dec 08 '20 at 17:45
  • @SAT I just did the proof for $a > 0$ because it works analogously when $a < 0$. Furthermore, if $a > 0$, $a = | a |$. – ViktorStein Dec 09 '20 at 06:49
  • Thank you for your reply. Could you elaborate on why we are using the absolute value in case of positive a? – Karina Dec 09 '20 at 08:33
  • @SAT It is not necessary, but I wanted it to look exactly like the identity in question. – ViktorStein Dec 09 '20 at 08:35
  • But if we merge two cases into one having 1/|a|, can we still do the change of variable in the integral? – Karina Dec 09 '20 at 08:44
  • I have created one question, I would be very grateful if you could have a look, since I feel stuck by the usage of absolute value. https://math.stackexchange.com/questions/3941181/how-to-approach-change-of-variable-in-integral-in-hax-hx-loga-scal – Karina Dec 09 '20 at 09:48