0

Knowing that $$H(X)=-\sum_{x,y} p(x,y) \log p(x)$$ and $$H(Y)=-\sum_{x,y} p(x,y) \log p(y)$$

Show that $H(X,Y)\le H(X)+H(Y)$

This is what I have solved so far, by using theory, I can simply say that $H(X,Y)=H(X)+H(Y\mid X)$ and therefore, $H(Y\mid X) \le H(Y)$. This is true and when $X$ and $Y$ are independent, then $H(Y\mid X) = H(Y)$.

But, How can I show it mathematically? For the RHS, I can just simply sum the two equation and having $-\sum_{x,y} p(x,y) (\log p(x)+ \log p(y))$. How does this helped in proving the inequality? Thanks.

Michael Hardy
  • 1
Data HV
  • 1

1 Answers1

0

So you want to show $-\sum_{x,y} p(x,y)\log p(x,y) \le -\sum_{x,y} p(x,y)[\log p(x)+\log p(y)]$. This is same as $\sum_{x,y} p(x,y) \log \frac{p(x)p(y)}{p(x,y)} \le 0$. But Jensen imples LHS $\le \log(\sum_{x,y} p(x,y)\frac{p(x)p(y)}{p(x,y)}) = 0$, as desired.

mathworker21
  • 34,399