1

I'm trying to show that $\hat{e}$ and $\hat{\beta}$, the residuals and coefficient vector for the OLS problem $y=X\beta+\epsilon$, are uncorrelated.

The proof I see online doesn't make sense to me because it doesn't subtract $E[\hat{e}]E[\hat{\beta}]$ when computing the covariance.

From page 21 of http://homepage.ntu.edu.tw/~ckuan/pdf/et01/et_Ch3.pdf it says $$ cov(\hat{e},\hat{\beta}) = E[(I-P)yy'X(X'X)^{-1}] \\ = (I-P)E[yy']X(X'X)^{-1} = \sigma_0^2(I-P)X(X'X)^{-1} = 0. $$ where $P=X(X'X)^{-1}X'$ is the orthogonal projection onto $span(X)$. This seems wrong to me because $cov(A,B)=E[AB^{T}]-E[A]E[B^T]$, but above is just $E[\hat{e}\hat{\beta}^T].$

Kashif
  • 1,497

1 Answers1

1

The vector $\hat e$ of residuals has expectation zero. Here's the proof: By definition, $$\hat e:=y-\hat y=y-X\hat \beta_T.$$ Recall Theorem 3.4(a) stated that $$ E(\hat\beta_T)=\beta_0 $$ where $\beta_0$ is such that $E(y)=X\beta_0$, as specified in Classical Condition [A2] (section 3.2.1). Since $X$ is non-stochastic (condition [A1]), it follows that $$ E(X\hat\beta_T)=XE(\hat\beta_T)=X\beta_0 = E(y). $$ Or you can prove this directly using the definition $\hat\beta_T:=(X'X)^{-1}X'y$: $$E(X\hat\beta_T)=E(X(X'X)^{-1}X'y)\stackrel{[A1]}=X(X'X)^{-1}X'E(y)\stackrel{[A2]}=X(X'X)^{-1}X'X\beta_0=X\beta_0=E(y).$$

grand_chat
  • 38,951
  • Thanks! It’s just so weird that this is never mentioned in books. I made an earlier post about this but no one answered so thanks a ton for your help/clarification! – Kashif Oct 26 '18 at 19:36
  • https://math.stackexchange.com/questions/2972111/mean-of-residuals-in-ols-is-0 – Kashif Oct 26 '18 at 19:37
  • @Glassjawed You are very welcome. The fact that the residuals have expectation zero is usually hiding somewhere in the typical textbook, often in the 'sampling properties' or 'theory' section. But sometimes it's not spelled out explicitly. – grand_chat Oct 26 '18 at 23:01