Statistics - Expectation of OLS residual squared

Question

Suppose that $y = \beta_0+\beta_1x + \epsilon$ where $\operatorname{E}[\epsilon\mid X]=0$ and $\operatorname{Var}[\epsilon\mid X] = \sigma^2$.

In what special case is $\hat{\epsilon_i}^2$ an unbiased estimator for $\sigma^2$? Is $\hat{\epsilon_i}^2$ a consistent estimator for $\sigma^2$?

I have proved that $$E[\hat{\epsilon_i}^2] = \sigma^2\left(1-\frac{1}{n}- \frac{(x_i -\bar{x})^2}{\sum_{j=1}^n (x_j -\bar{x})^2} \right)$$ But I don't know how it can be unbiased or maybe I have proved wrong?

My proof: \begin{equation} \begin{split} \nonumber \hat{\epsilon_i} & = y_i - \hat{y_i} \\ & = \beta_0 + \beta_1 x_i + \epsilon_i - \hat{\beta_0} - \hat{\beta_1}x_i \\ & = \epsilon_i + (\beta_0 - \hat{\beta_0}) + (\beta_1 - \hat{\beta_1 })x_i \\ \hat{\beta_0} &= \bar{y} - \hat{\beta_1} \bar{x} \\ & = \frac{1}{n} \sum_{i=1}^{n}(\beta_0 + \beta_1 x_i + \epsilon_i) - \hat{\beta_1} \bar{x} \\ & = \beta_0 + (\beta_1 - \hat{\beta_1})\bar{x} + \frac{1}{n} \sum_{i=1}^{n} \epsilon_i \\ \hat{\beta_1} &= \frac{\sum (x_i - \bar{x}) y_i}{\sum (x_i - \bar{x})^2} \\ & = \frac{\sum (x_i - \bar{x})(\beta_0 + \beta_1 x_i + \epsilon_i)}{\sum (x_i - \bar{x})^2} \\ & = \beta_1 + \frac{\sum (x_i - \bar{x})\epsilon_i}{\sum (x_i - \bar{x})^2} \\ \therefore \hat{\epsilon_i} &= \epsilon_i - \frac{1}{n}\sum_{j=1}^{n} \epsilon_j + (\beta_1 - \hat{\beta_1})(x_i - \bar{x}) \\ & = \epsilon_i - \bar{\epsilon} - \frac{\sum_{j=1}^{n} (x_j - \bar{x})\epsilon_j}{\sum_{j=1}^{n} (x_j - \bar{x})^2} (x_i - \bar{x}) \\ \therefore \hat{\epsilon_i}^2 & = \epsilon_i^2 + \bar{\epsilon}^2 +\frac{(x_i -\bar{x})^2}{[\sum_{j=1}^{n} (x_j -\bar{x})^2]^2} [\sum_{j=1}^{n} (x_j -\bar{x}) \epsilon_j]^2 \\ & -2 \epsilon_i \bar{\epsilon} - 2\epsilon_i \frac{x_i - \bar{x}}{\sum_{j=1}^{n}(x_j - \bar{x})^2} \sum_{j=1}^{n} (x_j - \bar{x}) \epsilon_j + 2 \bar{\epsilon} \frac{x_i - \bar{x}}{\sum_{j=1}^{n}(x_j - \bar{x})^2} \sum_{j=1}^{n} (x_j - \bar{x}) \epsilon_j \\ \end{split} \end{equation} Now take expectation of the six parts of $\hat{\epsilon_i}^2$ given X (conditional symbol is omitted): \begin{equation} \begin{split} \nonumber (1) \ E[\epsilon_i^2] &= \sigma^2 \\ (2) \ E[\bar{\epsilon}^2] &= \frac{\sigma^2}{n} \\ (3) \ E[\cdot] & = \frac{(x_i -\bar{x})^2}{[\sum_{j=1}^{n} (x_j -\bar{x})^2]^2} E\left[[\sum_{j=1}^{n} (x_j -\bar{x}) \epsilon_j]^2 \right] \\ &= \frac{(x_i -\bar{x})^2}{[\sum_{j=1}^{n} (x_j -\bar{x})^2]^2} E\left[ \sum_{j=1}^{n}(x_j-\bar{x})^2 \epsilon_j^2 + 2 \sum_{j=1}^{n-1} \sum_{k=j+1}^{n} (x_j-\bar{x})\epsilon_j (x_k-\bar{x})\epsilon_k \right] \\ & = \frac{(x_i -\bar{x})^2 \sigma^2}{\sum_{j=1}^{n} (x_j -\bar{x})^2} \\ (4) \ E[-2\epsilon_i \bar{\epsilon}] & = -\frac{2}{n} E[\epsilon_i \sum_{j=1}^{n} \epsilon_j] \\ & = -\frac{2\sigma^2}{n} \\ (5) \ E[\cdot] &= -2 \frac{x_i - \bar{x}}{\sum_{j=1}^{n}(x_j - \bar{x})^2} E\left[ \epsilon_i \sum_{j=1}^{n} (x_j - \bar{x}) \epsilon_j\right] \\ & = -2 \frac{(x_i - \bar{x})^2 \sigma^2}{\sum_{j=1}^{n}(x_j - \bar{x})^2} \\ (6) \ E[\cdot] &= 2 \frac{x_i - \bar{x}}{\sum_{j=1}^{n}(x_j - \bar{x})^2} E\left[ \bar{\epsilon} \sum_{j=1}^{n} (x_j - \bar{x}) \epsilon_j\right] \\ & = 2 \frac{x_i - \bar{x}}{\sum_{j=1}^{n}(x_j - \bar{x})^2} \frac{1}{n} E[\sum_{j=1}^{n} (x_j -\bar{x}) \epsilon_j^2] \\ & = 2 \frac{x_i - \bar{x}}{\sum_{j=1}^{n}(x_j - \bar{x})^2} \frac{1}{n} \sigma^2 \sum_{j=1}^{n} (x_j -\bar{x}) \\ & = 0 \\ \therefore E[\hat{\epsilon_i}^2] & = \sigma^2 - \frac{\sigma^2}{n} - \frac{(x_i -\bar{x})^2 \sigma^2}{\sum_{j=1}^{n} (x_j -\bar{x})^2} \\ & = \sigma^2\left(1-\frac{1}{n}- \frac{(x_i -\bar{x})^2}{\sum_{j=1}^{n} (x_j -\bar{x})^2} \right) \end{split} \end{equation}

Answer 1

You have proved (correctly IMHO) that $\texttt{E}(\hat{\epsilon}_i^2) \leq \sigma^2(1 - \frac{1}{n}) < \sigma^2,$ when ${\bf x_i}$'s are not all the same so ${\bf \sum_{i} (x_i - \bar{x})^2 > 0}$, which means $\hat{\epsilon}_i^2$ cannot be unbiased when $x_i$'s are not identical. For consistency you have to make some assumptions about $x_i$'s as $n \to \infty$. If you assume for example $\sum_{i=1}^n(x_i - \bar{x}_n)^2 \to \infty$ (where $\bar{x}_n = \dfrac{\sum_i^n x_i}{n}$). Then clearly keeping $i$ fixed and letting $n \to \infty$ $\texttt{E}(\hat{\epsilon_i}^2) \to \sigma^2.$

Why I think your computation is correct for the case when $x_i$'s are not the same lies below.

Consider the usual linear model setting.

$Y = X\beta + \epsilon$ where $\epsilon \sim N(0,\sigma^2I).$

$\hat{\epsilon} = Y - \hat{Y} = (I - P)Y = (I-P)(Y-X\beta).$ Where $P$ is the projection operator into the column space of $X$.

We have $$ \begin{align}\texttt{E}(\hat{\epsilon}{\hat\epsilon}^T) &= \texttt{E}((I-P)(Y-X\beta)(Y-X\beta)^T(I-P))\\ &=(I-P)\texttt{E}((Y-X\beta)(Y-X\beta)^T)(I-P))\\ &=\sigma^2 (I - P).\end{align}$$

So $\texttt{E}(\hat\epsilon_i^2) = \sigma^2(1 - p_{ii}),$ where $P=(p_{ij}).$

In this particular case $X = \begin{pmatrix} 1 & x_1 \\ 1 & x_2 \\ \vdots & \vdots\\ 1 & x_n \end{pmatrix}$ and $$P = UU^T$$ where $U = \begin{pmatrix}\dfrac{1}{\sqrt{n}} & \dfrac{x_1 - \bar{x}}{\sqrt{\sum_{i=1}^n(x_i-\bar{x})^2}} \\ \dfrac{1}{\sqrt{n}} & \dfrac{x_2 - \bar{x}}{\sqrt{\sum_{i=1}^n(x_i-\bar{x})^2}} \\ \vdots & \vdots & \vdots\\ \dfrac{1}{\sqrt{n}} & \dfrac{x_n - \bar{x}}{\sqrt{\sum_{i=1}^n(x_i-\bar{x})^2}} \end{pmatrix}$ since the columns of $U$ form an orthonormal basis of column space of $X.$

So $p_{ij} = \dfrac{1}{n} + \dfrac{ (x_i - \bar{x})(x_j - \bar{x}) }{\sum_{i=1}^{n}(x_i - \bar{x})^2}$ and $\texttt{E}(\hat\epsilon_i^2) = \sigma^2(1 - p_{ii}) = \sigma^2\left(1 - \dfrac{1}{n} - \dfrac{ (x_i - \bar{x})^2 }{\sum_{i=1}^{n}(x_i - \bar{x})^2}\right).$

However if $x_i$'s are identical. Then $P=\dfrac{{\bf 11}^T}{n}$ ( here ${\bf 1}$ denotes a vector of 1's) and even in this case $\texttt{E}(\hat\epsilon_i^2) = \sigma^2(1-\dfrac{1}{n}) < \sigma^2.$ So $\hat\epsilon_i^2$ is always biased.