Why does marginalization of a joint probability distribution use sums?

Question

I'm going through a book that talks about probability distributions. The part which is tripping me up is conceptual. It says:

"We can recover the probability distribution of any single variable from a joint distribution by summing (discrete case) or integrating (continuous case) over all the other variables."

Why is this? What does this mean? I know this is a very open ended question - but I really don't understand at a conceptual level what this means.

To be a bit more clear: Imagine a joint distribution Pr(x, y) of two discrete distributions Pr(x) and Pr(y) as a Hinton diagram - where Pr(x) is horizontally and Pr(y) is vertically represented. Say there are 5 values in both directions.

To get x then according to the above statement - do I add the first column and that would give me the first value of Pr(x) and so on? If so, why? If not, what is the right way?

Answer 1

Suppose $X$ can be either $1$, $2$, $3$ or $4$, and $Y$ can be either $1$, $2$, or $3$. What is $\Pr(X=1)$? It is a marginal probability. And it is \begin{align} \Pr(X=1) & = \Pr \Big( (X=1 \text{ and } Y=1)\text{ or }(X=1 \text{ and } Y=2) \text{ or }(X=1 \text{ and } Y=3) \Big) \\[10pt] & = \Pr(X=1\ \text{and } Y=1) + \Pr(X=1\ \text{and } Y=2) + \Pr(X=1\ \text{and } Y=3). \end{align} This is a sum of values of the joint probability distribution.

Answer 2

If $I$ is countable and rv $Y$ only takes values in $\{y_i\mid i\in I\}$ then: $$\{X=x\}=\bigcup_{i\in I}\{X=x\wedge Y=y_i\}$$ The sets $\{X=x\wedge Y=y_i\}$ are disjoint so consequently: $$P\{X=x\}=\sum_{i\in I}P\{X=x\wedge Y=y_i\}$$

If $Y$ is continuously distributed then the sum becomes an integral.

This also works if more generally $\{X=x\}$ is replaced by $\{X\in A\}$

Answer 3

First, consider the definition of the probability distribution or cumulative distribution function or simply distribution function of a one-dimensional random variable or single event as

The probability of the event $\{X\leq x\}$ is called a probability distribution of random variable $X$ and is denoted by $F_{X}(x)$ and stated as:

$$ F_{X}(x)=P(X\leq x) \hspace{1cm} for -\infty\leq x \leq \infty $$ In other words $F_{X}(x)$ is the probability that $X$ takes any value in the range $(-\infty,x)$.

Now suppose A and B are events defined by the correspondence $ A =\{X|X \leq x\}$ and $ B =\{Y|Y \leq y\}$ then genetically A corresponds to the set of all points to the left of the vertical line $X=x$ and B corresponds to the set of points below horizontal line $Y=y$.If $ A\cap B$ is the intersection of the two events then it's mapping will be a common area as shown in the fig here

Now consider the definition of the probability distribution or Distribution function for the two-dimensional random variable $(X, Y)$ as

The probability of the joint event $\{X\leq x, Y\leq y\}$ is called a Joint probability distribution of random variable $X$ and $Y$ is denoted by $ F_{X,Y}(x,y)$ and stated as:

$$ F_{X,Y}(x,y)=P(X\leq x,Y\leq y) $$ In other words if the events defined as above i.e. if $ A =\{X|X \leq x\}$ and $ B =\{Y|Y \leq y\}$ then $ A \cap B = \{(X,Y)|X\leq x,Y\leq y\}$ and we have $$ F_{X,Y}(x,y)=P(X\leq x,Y\leq y)=A \cap B $$ Now when we set $y$ to $\infty$,this is equivalent to making $B$ a certain event i.e. Y taking any from $-\infty$ to $\infty$ i.e. $ B =\{Y|Y \leq \infty\}=S$. where $S$ is sample space.So we have $A\cap B =A\cap S=A$

$\therefore \hspace{0.5cm} F_{X,Y}(x,\infty)=P(X\leq x,Y\leq \infty) \\= P(A \cap S) = P(A)\\= P(X \leq x) = F_{X}(x)$

That is nothing but the Distribution Function or Probability Distribution for the single event $A$

Now consider in terms of the simple definition of the Joint Probability

$$ P(A/B)=\frac{card(A \cap B)/card(S)}{car(B)/card(S)}=P(A \cap B)/P(B) $$

Again with the same reason if we Marginalized or made powerless to $B$ as $B =\{Y|Y \leq \infty\}=S$ we left with $P(A)$.