2

How to show that the entropy $H(\operatorname{Pois}(\lambda))$ of a Poisson distribution $\operatorname{Pois}(\lambda)$ is Concave in parameter $\lambda$? i-e

$$f(\lambda)\equiv H(\operatorname{Pois}(\lambda))=-\displaystyle \sum_{x=0}^{\infty} \Big( \frac{e^{-\lambda} \lambda^x}{x!} \Big) \cdot \text{log}_2\Big( \frac{e^{-\lambda} \lambda^x}{x!} \Big) $$ is Concave in $\lambda$?

My Thoughts:

  1. Second derivative Check : It turns out that $\frac{e^{-\lambda} \lambda^x}{x!}$ is log-concave in $\lambda$, since $\frac{d^2}{d \lambda^2}\text{log}_2\Big( \frac{e^{-\lambda} \lambda^x}{x!}\Big)=-\frac{x}{\lambda^2}$. So, each term of the above summation is (at-least) Log-concave because product of log-concave functions result in log-concave function. However, the total summation above is not necessarily log-concave since sum of log-concave functions is not guaranteed to be log-concave (but sum of concave functions is concave, though).

  2. Simulation: Graphical simulations suggest $f(\lambda)$ to be concave in $\lambda$.

  3. Log sum inequality might be helpful, but I don't see a very clear way to use it to prove concavity.

kaka
  • 1,896

1 Answers1

2

I am assuming base of the logarithm is $e$ instead of $2$ as one can easily convert the base to $e$ by taking a positive constant out.

Note that $f(\lambda)=E_\lambda(\log p_\lambda(X))$ where $p_\lambda$ is the pmf of $Poisson(\lambda)$ distribution and $X\sim Poisson(\lambda)$.

Then using the standard regularity conditions as Poisson belongs to Exponential family, $f''(\lambda)=\dfrac{d^2}{d\lambda^2}E_\lambda(\log p_\lambda(X))=E_\lambda(\dfrac{\partial^2}{\partial \lambda^2}\log p_\lambda(X))=-E_\lambda((\dfrac{\partial}{\partial\lambda}\log p_\lambda(X))^2)\leq0$

Landon Carter
  • 12,906
  • 4
  • 32
  • 79
  • Cool ! It looks similar to a Cramer-Rao bound stuff. How did you change the expectation from $E_x$ to $E_\lambda$? Any reference to the regularity conditions you used will be helpful. Many thanks. – kaka Dec 05 '16 at 22:24
  • 1
    The change in expectation is just a change of notation. Nothing to worry about there. I am using the Cramer Rao bound properties. For regularity conditions, see here: https://en.wikipedia.org/wiki/Cram%C3%A9r%E2%80%93Rao_bound#Regularity_conditions Basically they tell you that you can do what you like to do, as long as you are in an exponential family. – Landon Carter Dec 06 '16 at 16:57
  • On a different note, I noted you recently posted questions on Information Theory, Entropy and Coding. Can you please suggest a good book for this? At an introductory level, of course. – Landon Carter Dec 06 '16 at 17:01
  • "Elements of Information Theory " by Thomas Cover, is mostly I refer to. But I don't see it has discussed the concavity of information theoretic elements in parameter spaces of distributions. If you know any other book that is more relevant, it will be great. – kaka Dec 06 '16 at 18:46
  • 1
    Okay. I shall inform you if I find something nice. – Landon Carter Dec 07 '16 at 05:01
  • One thing that I missed in question is the $-$ sign in $H$. Would that result in $f''(\lambda)=-\dfrac{d^2}{d\lambda^2}E_\lambda(\log p_\lambda(X))=-E_\lambda(\dfrac{\partial^2}{\partial \lambda^2}\log p_\lambda(X))=E_\lambda((\dfrac{\partial}{\partial\lambda}\log p_\lambda(X))^2)\geq0$? But then $f(\lambda)$ is convex instead. – kaka Dec 07 '16 at 08:56
  • 1
    Hmm. Interesting. I'll look into it and tell you. Yes I searched a bit and found this: https://www.princeton.edu/~verdu/reprints/shannonpoissonOct2008.pdf You may go through it. I did not check if they PROVED log-concavity. But I saw they had graphed the function and it is log-concave. But they are saying something like "ultra-log-concavity" and not just log-concavity. Please go through that. In the meanwhile, I will think over what went wrong here. – Landon Carter Dec 08 '16 at 02:33
  • 1
    Yup, got the error. What I wrote is completely incorrect. The regularity conditions DO NOT tell us we can interchange derivative and expectation for $\log(p(X))$. The answer is therefore wrong. I will try to provide a correct solution. I am very sorry. – Landon Carter Dec 08 '16 at 12:02
  • 1
    No problem, thanks for your follow up and interest. I am deselecting your accepted answer now; to encourage other users to this question. – kaka Dec 09 '16 at 12:49
  • Start a bounty, if possible. I think this question needs attention. – Landon Carter Dec 13 '16 at 04:59