How can I prove the variance of residuals in simple linear regression?

Question

How can I prove the variance of residuals in simple linear regression?

Please help me.

$ \operatorname{var}(r_i)=\sigma^2\left[1-\frac{1}{n}-\dfrac{(x_i-\bar{x})^2}{\sum_{l=1}^{n}(x_l-\bar{x})}\right]$

I tried..

using $r_i=y_i-\hat{y_i}$

$\operatorname{var}(r_i)=\operatorname{var}(y_i-\hat{y_i})=\operatorname{var}(y_i-\bar{y})+\operatorname{var}(\hat{\beta_1}(x_i-\bar{x}))-2\operatorname{Cov}((y_i-\bar{y}),\hat{\beta_1}(x_i-\bar{x}))$

How can I go further?

If there's more information needed, please ask me to provide it.

https://stats.stackexchange.com/questions/115011/in-simple-linear-regression-where-does-the-formula-for-the-variance-of-the-resi?noredirect=1&lq=1. — StubbornAtom, Sep 25 '19 at 13:50
Your first expression seems to be missing a $,^2$ in the denominator — Henry, Apr 18 '20 at 01:41

score 2 · Answer 1 · answered May 08 '14 at 08:13

2

I believe the previous answer posted is incorrect, since $y_i$ and $\hat y_i$ are not uncorrelated. I would prove this as follows:

$\begin{align} \text{Cov}(r) &= \text{Cov}(y - Py), \quad P = X(X^TX)^{-1}X^T \\ & = \text{Cov}((I_n-P)y) \\ & = (I_n - P)\ \text{Cov}(y)\ (I_n - P)^T \\ & = (I_n-P)\ \sigma^2 I_n\ (I_n - P)^T \end{align}$

from which we can conclude that $\text{var}(r_i)=\sigma^2 (1 - P_{ii})$. It should be quite simple to confirm that your equation is recovered when you let $X$ be the matrix with a column of $1$'s (to represent $\bar x$) and a second column of the $x_i$'s.

answered May 08 '14 at 08:13

Brendon

121

This is not simple linear regression anymore since you are using vectors rather than scalars. – Fermat's Little Student Oct 01 '14 at 07:06
@Will, that is why I said "let X be the matrix with a column of 1's (to represent x¯) and a second column of the xi's." – Brendon Oct 01 '14 at 11:04

score 0 · Answer 2 · answered Jun 04 '13 at 15:07

0

Note that $$\begin{align}\operatorname{Var}(r_i) &=\operatorname{Var}(y_i-\hat{y_i}) \\ &=\operatorname{Var}(y_i)+ \operatorname{Var}(\hat{y_i}) \\ &= \sigma^2 + \operatorname{Var}(\bar y+\hat \beta _1(x_i-\bar x))\\ &= \sigma^2 +\operatorname{Var}(\bar y)+(x_i-\bar x)^2 \cdot\operatorname{Var}(\hat \beta _1)\\&=\sigma^2 +\dfrac{\sigma^2}{n}+\dfrac{\sigma^2 \cdot (x_i-\bar x)^2}{\sum_{i=1}^n(x_i-\bar x)^2}\\ &=\sigma^2 \left[1+\dfrac 1n +\dfrac{(x_i-\bar x)^2}{\sum_{i=1}^n(x_i-\bar x)^2} \right]\end{align}$$

answered Jun 04 '13 at 15:07

A.D

6,400
1
20
43

4

This incorrectly assumes that $y$ and $\hat{y}$ are uncorrelated. That this is not generally the case should be obvious (if they were uncorrelated, the slope would be 0). – Glen_b Sep 11 '14 at 00:39
Not only is the proof incorrect -- the formula you have derived is not correct and doesn't match the formula in the question. Terms 2 and 3 should be negative, not positive. – Denziloe Jan 26 '20 at 19:17

How can I prove the variance of residuals in simple linear regression?

2 Answers2