0

Let $X_1$ and $X_2$ be discrete random variables drawn according to probability mass function $p_1$ and $p_2$ over the respective alhabets $X_1={1,2,...m}$ and $X_2={m+1,...,n}$. Let $X=X_1$ with probability $a$ and $X=X_2$ with probability $(1-a)$. Quesiton: Find $H(X)$ in terms of $H(X_1)$ and $H(X_2)$ and $a$.

This problem is from Elements of Information Theory by Thomas M. Cover, Joy A. Thomas.

How to solve the problem ? Actually I am more confused why the answer is not simply $H(X)=-a*log(a) - (1-a)*log(1-a)$.

(The correct answer is $H(X)=-a*log(a) - (1-a)*log(1-a)+aH(X_1)+(1-a)H(X_2)$ according to the solution manual)

Thanks.

Jack2019
  • 1,525

2 Answers2

2

Let $Z$ be the random variable such that $Z=0$ if $X=X_1$ and $Z=1$ if $X=X_2$. To write it neatly: $H(X,Z) = H(X) + H(Z|X) = H(X)$ because $H(Z|X) = 0$.

Now write $H(X,Z) = H(Z) + H(X|Z) = H(Z) + p(Z=0) H(X|Z=0)+p(Z=1) H(X|Z=1)$

Now use $H(X|Z=0) = H(X_1)$,$H(X|Z=1) = H(X_2)$ in above equation to get: $H(X,Z) = H(Z) + H(X|Z) = H(Z) + a H(X_1)+(1-a) H(X_2)$

Balaji sb
  • 4,357
1

You have $P(X=i)=ap_1(i)$ for $i=1,\ldots,m$ so the contribution to the total entropy from the $X_1$ part equals $$-\sum_i ap_1(i)\log(ap_1(i)) = -a\log{a}\sum_i p_1(i) - a\sum_i p_1(i)\log{p_1(i)} = -a\log{a}+aH(X_1)$$ and then there is a similar term with $(1-a)$ and $X_2$.

Michal Adamaszek
  • 4,018