0

There exist a relationship between MI and entropy of two random variables i-e $$ I(X;Y)=H(Y)-H(Y|X).$$

But what if $ \overrightarrow X \: \in \mathbb {\{0,1\}}^2$ and $ \overrightarrow Y \: \in \mathbb R^{+ \: 3}$ be the random vectors. How to specify the mutual information $I(\vec X ; \vec Y)$ between them i-e ? $$I(\overrightarrow X \: ; \overrightarrow Y)= ?$$

I suspect that KL divergence may be one possible answer but then how to define it for vectors?

kaka
  • 1,896
  • 1
    A "random vector" is a "random variable". You seem to believe that a random variable is a scalar - in general, it doesn't need to be, it can be multidimensional - and in particular, in the MI definition, it can well be multidimensional – leonbloy Aug 12 '15 at 18:53
  • @leonbloy I know that random vector is a random variable. What I didn't explain in question was that how to expand $I( \overrightarrow X;\overrightarrow Y)=H(\overrightarrow X)-H(\overrightarrow X| \overrightarrow Y) $ in terms of components. (if $ \overrightarrow X$ is 2D and $\overrightarrow Y $ is in 3D) – kaka Aug 12 '15 at 21:00

1 Answers1

1

Mutual information applies to probability distributions. Since a random vector is still just a probabilities distribution (albeit one where the random variables are multi-dimensional) then it doesn't need any special treatment. You'd simply take each vector (e.g., $x=(0,0)$,$P(x=(0,0))=0.4$) as a possible value of the random variables $X$ and $Y$ and then add them up in the usual way for mutual information.

Sam Jaques
  • 2,090