0

What does it mean, exactly, when it is said that:

If two variables $x$ and $y$ are correlated with correlation coefficient $\rho >0$ then $\rho^2$ of the variation in $x$ can be explained by the variation in y.

Is it something like:

Suppose we have two lists of variables $x_i$ and $y_i$. Let the variance of $y$ be $\sigma_y^2$ and the correlation between $x$ and $y$ be $\rho.$ Consider the best-fit line of least squares regression between them. Let $z(x_i)$ be this line of best fit. Consider the list of points $z(x_i)-x_i$. The variance of this list of points will be $\rho^2 \sigma_y^2$ (it will be smaller).

Jbag1212
  • 1,435
  • 1
    Does this answer your question? Pearson Correlation Coefficient Interpretation – P.S. Dester Aug 04 '23 at 21:55
  • @P.S.Dester No, not really. I am asking about the oft repeated phrase, "the square of the correlation coefficient explains the percentage of the variance that is explained by another variable." That question is not addressing that particular phrase at all. – Jbag1212 Aug 04 '23 at 22:19
  • Where did you read that phrase? It is well known that in PCA the eigenvalues of the covariance (not correlation!) matrix are variances whose sum is the total variance of the sum of the independent factors. Hence, what follows about explained percentage by those single eigenvectors? BTW: a mathematician should refrain from using the phrase "something like". – Kurt G. Aug 05 '23 at 06:24
  • https://math.stackexchange.com/q/3076842/321264 – StubbornAtom Aug 05 '23 at 06:30
  • @KurtG. Something akin to that phrase is quite common in many referenes. The reason I use the phrase "something like" is that the precise nature of the phrase is rarely specified and often varies – Jbag1212 Aug 06 '23 at 04:12
  • @KurtG. See https://www.investopedia.com/terms/c/correlationcoefficient.asp "Nor does the correlation coefficient show what proportion of the variation in the dependent variable is attributable to the independent variable. That's shown by the coefficient of determination, also known as R-squared, which is simply the correlation coefficient squared." https://stats.stackexchange.com/questions/45419/do-correlation-or-coefficient-of-determination-relate-to-the-percentage-of-value , https://www.sportsci.org/resource/stats/correl.html "The resulting statistic is known as variance explained" – Jbag1212 Aug 06 '23 at 04:15

1 Answers1

2

You can express $\ x\ $ as the sum $$ x=\alpha y+z $$ of two random variables, one of which (viz. $\ \alpha y\ $) is directly proportional to $\ y\ $ and the other of which (viz. $\ z\ $) is uncorrelated to $\ y\ .$ To achieve this, you have to choose $\ \alpha=\frac{\rho\sigma_x}{\sigma_y}\ .$ The variance $\ \sigma_x^2\ $ of $\ x\ $ is now the sum $$ \sigma_x^2=\sigma_{\alpha y}^2+\sigma_z^2 $$ of a part (viz. $\ \sigma_{\alpha y}^2\ $) resulting from the "dependence" of $\ x\ $ on $\ y\ $, and a part (viz. $\ \sigma_z^2\ $) resulting from the "dependence" of $\ x\ $ on a variable uncorrelated to $\ y\ $. Since $\ \sigma_{\alpha y}^2=\rho^2\sigma_x^2\ $ and $\ \sigma_z^2=\big(1-\rho^2\big)\sigma_x^2\ $, then a proportion $\ \rho^2\ $ of the variance of $\ x\ $ comes from its "dependence" on $\ y\ $ and a proportion $\ 1-\rho^2\ $ from its "dependence" on a variable uncorrelated to $\ y\ $. I've put scare quotes around the word "dependence" here because the random variable $\ z\ $ is not necessarily independent of $\ y\ $. If the distribution of $\ (x,y)\ $ is bivariate normal, however, then $\ z\ $ and $\ y\ $ will be independent, and the scare quotes become unnecessary.

lonza leggiera
  • 28,646
  • 2
  • 12
  • 33
  • The variable z is not necessarily independent, but it is uncorrelated, correct? (The relation z=|y| neatly shows an example of this). – Jbag1212 Aug 06 '23 at 04:00
  • You say, "you can express $x$ as the sum." Does this always hold? Is that fact trivial, or does it need to be demonstrated? – Jbag1212 Aug 06 '23 at 04:02
  • 1
    @Jbag1212 That answer said that $z$ is uncorrelated in the first sentence and gave a hint how to achieve this always. Further hint: with the given $\alpha,,$ what is the covariance of $y$ and $z,?$ – Kurt G. Aug 06 '23 at 05:38
  • ... when you define $z$ as $z:=x-\alpha y,.$ – Kurt G. Aug 06 '23 at 05:51