Correlation Coefficient and Determination Coefficient

Question

I'm new to linear regression and am trying to teach myself.

In my textbook there's a problem that asks "why is $R^{2}$ in the regression of $Y$ on $X$ equal to the square of the sample correlation between X and Y?"

I've been throwing my head against this for a while and I keep getting stuck because in the correlation coefficient there is a $X$ and $\bar{X}$ term, whilst in the $R^{2}$ term there is no such thing.

Can anyone provide a derivation as to why $R^{2}$ is the correlation coefficient squared?

Thanks!

Answer 1

Suppose that we have $n$ observations $(x_1,y_1),\ldots,(x_n,y_n)$ from a simple linear regression $$ Y_i=\alpha+\beta x_i+\varepsilon_i, $$ where $i=1,\ldots,n$. Let us denote $\hat y_i=\hat\alpha+\hat\beta x_i$ for $i=1,\ldots,n$, where $\hat\alpha$ and $\hat\beta$ are the ordinary least squares estimators of the parameters $\alpha$ and $\beta$. The coefficient of the determination $r^2$ is defined by $$ r^2=\frac{\sum_{i=1}^n(\hat y_i-\bar y)^2}{\sum_{i=1}^n(y_i-\bar y)^2}. $$ Using the facts that $$ \hat\beta=\frac{\sum_{i=1}^n(x_i-\bar x)(y_i-\bar y)}{\sum_{i=1}^n(x_i-\bar x)^2} $$ and $\hat\alpha=\bar y-\hat\beta\bar x$, we obtain \begin{align*} \sum_{i=1}^n(\hat y_i-\bar y)^2 &=\sum_{i=1}^n(\hat\alpha+\hat\beta x_i-\bar y)^2\\ &=\sum_{i=1}^n(\bar y-\hat\beta\bar x+\hat\beta x_i-\bar y)^2\\ &=\hat\beta^2\sum_{i=1}^n(x_i-\bar x)^2\\ &=\frac{[\sum_{i=1}^n(x_i-\bar x)(y_i-\bar y)]^2\sum_{i=1}^n(x_i-\bar x)^2}{[\sum_{i=1}^n(x_i-\bar x)^2]^2}\\ &=\frac{[\sum_{i=1}^n(x_i-\bar x)(y_i-\bar y)]^2}{\sum_{i=1}^n(x_i-\bar x)^2}. \end{align*} Hence, \begin{align*} r^2 &=\frac{\sum_{i=1}^n(\hat y_i-\bar y)^2}{\sum_{i=1}^n(y_i-\bar y)^2}\\ &=\frac{[\sum_{i=1}^n(x_i-\bar x)(y_i-\bar y)]^2}{\sum_{i=1}^n(x_i-\bar x)^2\sum_{i=1}^n(y_i-\bar y)^2}\\ &=\biggl(\frac{\sum_{i=1}^n(x_i-\bar x)(y_i-\bar y)}{\sqrt{\sum_{i=1}^n(x_i-\bar x)^2\sum_{i=1}^n(y_i-\bar y)^2}}\biggr)^2. \end{align*} This shows that the coefficient of determination of a simple linear regression is the square of the sample correlation coefficient of $(x_1,y_1),\ldots,(x_n,y_n)$.

Answer 2

The complete proof of how to derive the coefficient of determination $R^{2}$ from the Squared Pearson Correlation Coefficient between the observed values $y_{i}$ and the fitted values $\hat{y}_{i}$ can be found under the following link:

http://economictheoryblog.wordpress.com/2014/11/05/proof/

In my eyes it should be pretty easy to understand, just follow the single steps.

Answer 3

There are many forms of the computation available online (such as the Wikipedia page on the correlation coefficient http://en.wikipedia.org/wiki/Pearson_product-moment_correlation_coefficient#Pearson.27s_correlation_and_least_squares_regression_analysis ) but note that this is a magical algebraic property of least squares linear regression, not linear regression in general.

Answer 4

There are different forms to express R2: Some expressions have (X-Xbar) squared in the numerator, while others express it just with the square of predicted ys. All forms are equivalent.

References: Dougherty; Gujarati; Wooldridge

Answer 5

The following answer assumes you are familiar with linear algebra and matrix factorization. The basic idea is to express both the Pearson correlation and $R^2$ in terms of vector of observations (and intercept).

Suppose you have made $n$ observations $(x_i, y_i): i = 1, 2, \ldots, n$, denote the column vector $(y_1, y_2, \ldots, y_n)'$ and the column vector $(x_1, x_2, \ldots, x_n)'$ by $y$ and $x$ respectively. Moreover, denote the $n$-long column vector of all ones by $e$. The design matrix $X$ thus can be written as $X = \begin{bmatrix} e & x \end{bmatrix}$. Let $X = QR$ be the QR decomposition of $X$, where $Q = \begin{bmatrix} q_1 & q_2\end{bmatrix}$ is column-wise orthogonal, i.e., $q_1'q_1 = q_2'q_2 = 1$ and $q_1'q_2 = 0$.

With these preparations, it is readily seen that the Pearson correlation $r$ is \begin{align} r = \frac{\sum(x_i - \bar{x})(y_i - \bar{y})}{\sqrt{\sum(x_i - \bar{x})^2\sum(y_i - \bar{y})^2}} = \frac{x'(I - q_1q_1')y}{\sqrt{x'(I - q_1q_1')x \cdot y'(I - q_1q_1')y}}. \tag{1} \end{align} While the coefficient of determination is given by \begin{align} R^2 = \frac{\sum(\hat{y}_i - \bar{y})^2}{\sum(y_i - \bar{y})^2} = \frac{y'(H - q_1q_1')y}{y'(I - q_1q_1')y} = \frac{(y'q_2)^2}{y'(I - q_1q_1')y}, \tag{2} \end{align} where $H = X(X'X)^{-1}X' = QQ' = q_1q_1' + q_2q_2'$ is the so-called hat matrix. It thus follows by $(1)$ and $(2)$ that to show $R^2 = r^2$, it is sufficient to prove \begin{align} (x'(I - q_1q_1')y)^2 = (y'q_2)^2 \times x'(I - q_1q_1')x. \tag{3} \end{align} Since $x$ is in the space spanned by two orthonormal vectors $q_1$ and $q_2$, we have $x = (x'q_1)q_1 + (x'q_2)q_2$, whence \begin{align} & x'(I - q_1q_1')x = \|x\|^2 - (x'q_1)^2 = (x'q_2)^2, \\ & x'(I - q_1q_1')y = ((x'q_1)q_1' + (x'q_2)q_2')(y - q_1q_1'y) = (x'q_2)q_2'y. \end{align} Hence $(3)$ holds. This completes the proof.