Kullback-Leibler divergence of two exponential distributions with different scale parameters

Question

The question is as follows:

"Calculate the Kullback-Leibler divergence between two exponential distributions with different scale parameters. When is it maximal?"

I have tried something but I come to a wrong conclusion (at least comparing with Wikipedia).

Let the KL-divergence between the approximating distribution $p_\theta$ and the "true" distribution $p_{\theta_0}$ be defined as

$M(\theta) = P_{\theta_0} \log \frac{P_\theta}{P_{\theta_0}}$

The density of an exponential distribution is given by

$p_\theta(x) = \theta e^{-\theta x}$

hence for the "true" distribution we have

$p_{\theta_0}(x) = \theta_0 e^{-\theta_0 x}$

which gives

$P_{\theta_0}\log \frac{P_\theta}{P_{\theta_0}} = P_{\theta_0} \log \frac{\theta e^{-\theta x}}{\theta_0 e^{-\theta_0 x}}$

which we can simplify to

$P_{\theta_0}\log \frac{P_\theta}{P_{\theta_0}} = P_{\theta_0} \log(\theta) - \log(\theta_0) - (\theta - \theta_0)x$

(This is where I think I might have made a possible mistake)

Because $x$ under the true distribution is exponentially distributed with scale parameter $\theta_0$ its mean is given by $1/\theta_0$ and as such we find that the KL-divergence is

$M(\theta) = \log(\theta) - \log(\theta_0) - (\theta - \theta_0)\frac{1}{\theta_0} =\log(\theta) - \log(\theta_0) -\frac{\theta}{\theta_0}+1$

However, Wikipedia (link) gives the following KL-divergence for two exponential distributions

$M(\theta) =\log(\theta) - \log(\theta_0) +\frac{\theta}{\theta_0}-1$

My answer has the signs flipped, where did I make an error?

Answer 1

Let $p_\theta$ be the true distribution and $p_\phi$ be the approximating one. Then the KL divergence is \begin{align} \mathcal{D}_\text{KL}[p_\theta || p_\phi] &= \int_D p_\theta(x) \log\frac{p_\theta(x)}{p_\phi(x)} dx \\ &= \int_D \theta\exp(-\theta x)\log\frac{\theta\exp(-\theta x)}{\phi\exp(-\phi x)} dx \\ &= \int_D \theta\exp(-\theta x)\left[ \log(\theta)-\log(\phi)-\theta x + \phi x \right] dx \\ &= \log\frac{\theta}{\phi}\underbrace{\int_D \theta\exp(-\theta x) dx}_{1 \text{ by}\int_D p_\theta=1} + (\phi-\theta)\underbrace{\int_D \theta\exp(-\theta x) x\, dx}_{\mathbb{E[x]}=\theta^{-1}\text{ if } x\,\sim\, p_\theta} \\ &= \log\theta - \log\phi +\frac{\phi - \theta}{\theta} \\ &= \log\theta - \log\phi +\frac{\phi}{\theta} - 1 \\ \end{align} which is what Wikipedia gives.

As for "when is it maximal?" (kind of an odd question), we look at the derivative of the model parameter: $$ \frac{\partial}{\partial \phi} \mathcal{D}_\text{KL}[p_\theta || p_\phi] =\frac{\partial}{\partial \phi} (\log\theta - \log\phi +\frac{\phi}{\theta} - 1) = \frac{-1}{\phi} + \frac{1}{\theta} $$ which shows unsurprisingly the distance is minimal when $\theta=\phi$. Note that the distance grows without bound as $\phi\rightarrow\infty$ or $\phi\rightarrow 0$ (noting that $\phi > 0$ by definition). Interestingly, $\phi\rightarrow \infty$ implies $\partial_\phi \mathcal{D}_\text{KL}[p_\theta || p_\phi] \rightarrow 1/\theta$ whereas $\partial_\phi \mathcal{D}_\text{KL}[p_\theta || p_\phi] \rightarrow -\infty$ as $\phi\rightarrow 0$.

Hopefully I didn't make a mistake!

---

Note: be careful because there are "variants" of the KL that are closely related and easy to mis-type, e.g., $$ -\mathcal{D}_\text{KL}[p_\theta || p_\phi] = \int p_\theta(x) \left(-\log\frac{p_\theta(x)}{p_\phi(x)}\right) = \int p_\theta(x) \log\frac{p_\phi(x)}{p_\theta(x)} $$ $$ \mathcal{D}_\text{KL}[p_\phi|| p_\theta] = \int p_\phi(x) \log\frac{p_\phi(x)}{p_\theta(x)} $$