1

Say I have a sequence of probability measures $\mu_n$ with Lebesgue densities $f_n$ that converges in total variation norm to a probability measure $\mu$ with Lebesgue density $f$. Further assume that $f$ and $f_n$ are continuous and bounded, and that the entropy of each measure is finite. Can I conclude that the sequence of entropies converges to the entropy of the limiting measure? I.e. $$\int f_n(x) \log f_n(x) dx \to \int f(x)\log f(x)dx $$

I had the following idea, but now I'm stuck: Convergence in total variation norm implies weak convergence, thus by the Portmanteau Theorem we have for each fixed $k$ that $$F_{k,n} := \int f_n(x)\log f_k(x) dx \to \int f(x)\log f_k(x) dx.$$ But I've no idea if this helps with finding the limit of the "diagonal" $F_{n,n}$.

J. Curwen
  • 11

1 Answers1

1

No, not necessarily. As a counterexample, take $f_n(x)=(n-|x|)/(n^2a_n)$ for $|x|\leq n$, and with $f_n(x)=0$ for $|x|>n$. If we choose the sequence $a_n$ such that $a_n \to \infty$ then the measures $\mu_n$ converge to $0$ in total variation norm because $\int_{-n}^{n} f_n(x)\, dx=1/a_n$ (and $\|\mu_n\|_{TV}=\int_{-\infty}^{\infty} |f_n(x)|\, dx$). But the entropy is $\int_{-n}^{n} f_n(x) \ln(f_n(x)) dx$. (Some people use the negative of this definition.) Using the formula for $f_n$ gives the evaluation of the integral as $(-\frac{1}{2}-\ln(n)-\ln(a_n))/a_n$. So, if $\limsup_{n \to \infty} \ln(n)/a_n>0$ then the entropies do not converge to $0$, which is the entropy of the $0$ measure. For example, choose $a_n=\ln(n+1)$ (assuming $n$ starts at $1$). Then the entropies actually converge to $-1$.

Shannon Starr
  • 633
  • 1
    Thanks for your counterexample, @shannon-starr !. But I notice your $f_n$ are not probability densities. So I still wonder if works when all densities (including the one of the limiting measure) integrate to $1$. Multiplying all of your densities by $a_n$ does not result in a valid counterexample. – J. Curwen Dec 13 '21 at 07:20
  • 1
    Take $g_n = f_n + (1-\int f_n(x), dx) \phi$ for $\phi=\mathbf{1}_{[-1/2,1/2]}$. Note that the integrand for entropy will be altered for $x \in [-1/2,1/2]$ but the support for the $f_n$'s is increasing to $\infty$. The limit of the entropies of the $g_n$'s will be the entropy of $\phi$ plus $-1$: $H(\phi)-1$. But the entropy of the limit will be just the entropy of $\phi$: $H(\phi)$. Entropy is semi-continuous. The way it is written here $H$ is upper semi-continuous. It can jump up in the limit, but not down. Here it jumps up from $H(\phi)-1$ to $H(\phi)$. – Shannon Starr Dec 13 '21 at 15:49