3

Let the Kullback-Leibler (KL) divergence of two distributions $p(x)$ and $q(x)$ be defined as

$D(P||Q) = E_p(\log p(x) - \log q(x))$

Let

$p(x) \sim \text{Laplace}(\mu_1, b_1)$

and

$q(x) \sim \text{Laplace}(\mu_2, b_2)$

then

$ \begin{align} D(P||Q) &= E_p(\log p(x) - \log q(x)) \\ &= E_p\left( \log \frac{b_2}{b_1} - \frac{|x-\mu_1|}{b_1} + \frac{|x-\mu_2|}{b_2}\right)\\ &= \log \frac{b_2}{b_1} - \frac{1}{b_1}E_p|x-\mu_1| + \frac{1}{b_2}E_p|x-\mu_2|\\ \end{align} $

The first term is a constant.

The second term is the median divided by $b_1$ (IM NOT SURE ABOUT THIS STEP)

The third term is given by $b_1/b_2$, because if

$X\sim\text{Laplace}(\mu_1,b_1)$

then

$X-c\sim\text{Laplace}(\mu_1-c,b_1)$

from which follows that

$|X-c|\sim\text{exp}(b_1^{-1})$

Hence the Kullback-Leibler divergence of two Laplace distributions is given by

$ \begin{align} D(P||Q) &= E_p(\log p(x) - \log q(x)) \\ &= \log \frac{b_2}{b_1} - 1 + \frac{b_1}{b_2} \end{align} $

I couldnt find the correct answer anywhere online so I was wondering if anyone here knows the right answer.

Rainymood
  • 181

1 Answers1

1

For the unsure term, you can apply the same reasoning as for the third to get that it is the expectation of an exponential distribution with parameter $b_2^{-1}$ and divide that mean by $b_2$.

P. Quinton
  • 6,031