9

Can conditional distributions determine the joint distribution?

For example, let $X_1, \dots, X_n$ be random variables.

Can their joint distribution be determined from the conditional distribution of $X_i$ given others, $i=1, \dots, n$?

Can their joint distribution be determined from other types of conditional distributions, such as the type of $P(X_i|X_j)$, and/or the type of $P(X_i, X_j | \text{others})$, and/or other types?

Thanks!

Tim
  • 47,382
  • Except in the important case of independence, the answer to the first question is no. One can get sometimes useful inequalities. If more information is given, one can get the joint distribution. As a simple example, suppose that we are dealing with discrete distibutions, and we know $\Pr(X=a)$ and $\Pr(Y=b|X=a)$ for every $a$ and $b$. Then by the definition of conditional probability, we know $\Pr((X= a)\cap (Y=b))$, that is, we know the joint distribution. Similar considerations hold for conditional densities. – André Nicolas Mar 04 '14 at 16:26
  • @AndréNicolas: Thanks! Here is how my questions arose. In Gibbs sampling, the conditional distributions of Xi given others, i=1,…,n are called full conditional distributions. I heard that the joint distribution can be determined by all the full conditionals, but I am not sure if that is correct. – Tim Mar 04 '14 at 21:34
  • 7
    The answer to the first question is yes, without independence hypothesis. When densities exist, for two random variables $X$ and $Y$, note that $$f_{X\mid Y}(x\mid y)f_Y(y)=f_{X,Y}(x,y)=f_{Y\mid X}(y\mid x)f_X(x)$$ hence, $$f_Y(y)\int\frac{f_{X\mid Y}(x\mid y)}{f_{Y\mid X}(y\mid x)}dx=\int f_X(x)dx=1$$ hence $$f_Y(y)=\left(\int\frac{f_{X\mid Y}(x\mid y)}{f_{Y\mid X}(y\mid x)}dx\right)^{-1}$$ and $f_{X,Y}$ is fully determined by $f_{X\mid Y}$ and $f_{X\mid Y}$ through the formula $$f_{X,Y}(x,y)=f_{X\mid Y}(x\mid y)\left(\int\frac{f_{X\mid Y}(x\mid z)}{f_{Y\mid X}(z\mid x)}dz\right)^{-1}.$$ – Did Apr 30 '16 at 13:45
  • 1
    @Did is the variable replacement in last line correct? – Alexander Gruber Feb 01 '18 at 17:56
  • 1
    @AlexanderGruber It is not, in the integral one should replace the content of the parenthesis by $$\int\frac{f_{X\mid Y}(z\mid y)}{f_{Y\mid X}(y\mid z)}dz$$ Well spotted, thanks. – Did Feb 01 '18 at 19:36

2 Answers2

5

This can in fact be true, and is very important and relevant in the context of Gibbs sampling as mentioned. If a joint distribution $P_{XY}$ has two conditional distributions $P_{X \mid Y}$ and $P_{Y \mid X}$, then knowledge of the conditional distributions alone is often sufficient determine $P_{XY}$. See Arnold and Press, 1989.

alex404
  • 61
  • A little more explanation with proper references is required especially when you are contradicting answers given by others. – Shailesh Apr 17 '16 at 09:45
  • This paper gives a counter-example where two conditional distributions with densities are incompatible in the sense that they do not describe a valid joint distribution. This seems to contradict the comment of @Did, unless one is careful about interpreting the phrase "when densities exist" to mean that suitable marginal distributions exist. – R Hahn Aug 02 '19 at 22:34
1

Well, I guess you have to know at least one density function among all the random variables $X_i$. Recall the basic property of conditional distributions that $P(X_i\in \mathrm{d}t_i, X_j\in \mathrm{d}t_j) = P(X_i\in \mathrm{d}t_i) P(X_j\in \mathrm{d}t_j| X_i\in \mathrm{d}t_i)$. And by induction you can figure out the joint distributions based on conditional distributions among the variables. Hope this helps!

Cjay
  • 44
  • 4