0

In the book "Elements of Information Theory" $H(Y\mid X)$ is defined like that and then it's shown that this is

\begin{align*} H(Y\mid X) &= \sum_{x\in X} p_X(x) H (Y\mid X = x) \\ &= - \sum_{x\in X} \sum_{y \in Y} p(x,y) \log_2 p(y\mid x) \\ &= \mathbb{E}\big[ - \log_2 p(Y\mid X) \big] \\ &= - \mathbb{E}\big[\log_2 p(Y\mid X) \big] \end{align*}

I fail to understand why $H(Y\mid X)$ is actually "defined" like that. What's the justification for this definition?

I think I'm confused because

\begin{align*} H(Y,X) = - \sum_{x\in X \\y\in Y} p(x,y) \log_2 p(x,y) \end{align*}

but

\begin{align*} H(Y\mid X) \neq - \sum_{x\in X \\y\in Y} p(y\mid x) \log_2 p(y\mid x) \end{align*}

and I don't see why.

Stefan Falk
  • 1,217
  • 1
  • 11
  • 24

2 Answers2

1

The question is a problem for me, ever. But now, I make sense about it.

  • First, this is a definition about conditional entropy. Different people can give different definitions. However, it must be reasonable for consistency.
  • Second, why the formula you mentioned is not reasonable? Use the formula following, $$H(Y|X=x) = -\sum_{x,y}P(y|x)\log(P(y|x))$$ because in this formula, $X=x$ is give as a constraint or precondition. for $H(Y|X)$, there is no any preconditions, that is to say, $X$ can be any value. so you need to take all $X=x$ into consideration. for each $X=x$, there exists a $P(x)$, so the definition for $H(Y|X)$ is more reasonable in this formula, $$H(Y|X) = -\sum_{x,y}P(x)P(y|x)\log(P(y|x)) = -\sum_{x,y}P(x,y)\log(P(y|x))$$

I hope this to make it clear for you.

Ernie060
  • 6,073
Limengyang
  • 11
  • 1
0

A random variable $Y$ has an entropy $H(Y)$, which gives you the amount of information (in bit), required to describe the random variable, on average. When using the conditional entropy, we have some extra information, described with random variable $X$, that is, hopefully, somehow correlated with $Y$. So, if, for example, $x_1 \in X$ happens, we would know that some outcomes of $Y$ are more probable. Assume $x_i\in X$ are possible outcomes of $X$. $H(Y|x_i)$ is the amount of information, required to exactly pinpoint an event in $Y$, on average, given that $x_i$ has happened. Finally, you have $H(Y|x_i)$ for all $x_i\in X$. Therefore, you should take the average of them, considering their probability of occurrence $p_X(x_i)$, to see how much information is required on average to pinpoint events of $Y$, given you have a stream of information from $X$.

Med
  • 2,530
  • It's the "you should take the average of them" that I don't really understand here. I mean, I get that it makes sense - but in this case it appears to come purely from its definition or the way it has been engineered. So is $H(Y|X)$ just "designed" this way? – Stefan Falk Nov 07 '16 at 17:08
  • I think I'm so confused because

    \begin{align} H(X,Y) = - \sum_{x\in X \y\in Y} p(x,y) \log_2 p(x,y) \end{align}

    but

    \begin{align} H(X\mid Y) \neq - \sum_{x\in X \y\in Y} p(x\mid y) \log_2 p(x\mid y) \end{align}

     – Stefan Falk Nov 07 '16 at 17:14
  • I think it is described well here https://en.wikipedia.org/wiki/Conditional_entropy – Med Nov 07 '16 at 18:21
  • Well, I've seen that. But it's also not really explaining why this is getting averaged. Why not take the absolute value or square it? Why not simply

    \begin{align} H(Y\mid X) =- \sum_{x\in X } \sum_{\y\in Y} p(y\mid x) \log_2 p(y\mid x) \end{align}

    but

    \begin{align} H(Y\mid X) =- \sum_{x\in X } p(x) \sum_{\y\in Y} p(y\mid x) \log_2 p(y\mid x) \end{align}

     – Stefan Falk Nov 07 '16 at 18:27
  • I am not sure if it is helpful, but when you do averaging using probabilities, the fractions should add up to one. So $P(x,y)$, for all possible $x$ and $y$, add up to one. However, it is not true for $P(y|x)$. If you add them, you get a number more than $1$. Does this help? – Med Nov 07 '16 at 20:44