3

For context,
the linear model is
$$\mathbf{y} = X\mathbf{\beta} + \mathbf{\varepsilon} $$ where $\mathbf{y}$ is the vector of responses $y_i,i=1,2,\ldots,n$, $X$ is the deterministic design matrix $n\times p$ and $\beta$ is the vector of parameters, $p\times 1$.
Also, $\varepsilon$ is the vector of uncorrelated errors with zero mean, $\varepsilon_i,i=1,2,\ldots,n$ where $\mathrm{Var}(\varepsilon_i) = \sigma^2.$

Now, the MLE of $\sigma^2$ in the multiple linear model is
$$\hat{\sigma}^2 := \frac{1}{n} (y-XB)^T (y-XB) $$
And I'm given the formula that for any random $k\times 1$ vector $u$, and constant $k\times k$ matrix $A$, we have

$$\mathbb{E}(u^T A u) = \mathrm{tr}(AV) + \mu^T A \mu$$ where $V=\mathrm{Var}(u), \ \mu = \mathbb{E}(u)$.

So I try to apply this to my MLE.
$$\hat{\sigma}^2 := \frac{1}{n} \varepsilon^T \varepsilon \\ \mathbb{E}(\hat{\sigma}^2) = \frac{1}{n} \mathbb{E}(\varepsilon ^T I_{n\times n} \varepsilon) $$
with $A = I_{n\times n}, u = \varepsilon$.
So $V = \mathrm{Var}(\varepsilon) = \sigma^2 I_{n\times n}, \mu = \mathbb{E}(\varepsilon) = \mathbf{0}$.
hence, $$\mathbb{E}(\hat{\sigma}^2) = \frac{1}{n} ( \mathrm{tr}(I_{n\times n}\sigma^2 I_{n\times n}) = \sigma^2.$$

However, this is incorrect as the solution says that it is $\frac{n-p}{n}\sigma^2$ and it also shows me how it's done. However, I'm not sure where I went wrong.

Natash1
  • 1,379

1 Answers1

1

The problem here is that even tough the error term $\epsilon$ has variance given by $\sigma^2 I_{n\times n}$ the residuals $\hat{\epsilon}$ have a different variance.

Note that, by definition

$\hat{\sigma}^2=\frac{1}{n}(Y-X\hat\beta)'(Y-X\hat\beta)=\frac{1}{n}\hat{\epsilon}'\hat{\epsilon}$

You should check that, $\hat\epsilon=(I_{n\times n}-P)Y$ where $P$ is the projection matrix $P:=X(X'X)^{-1}X'$. Hence, $Var(\hat\epsilon)=\sigma^2(I_{n\times n}-P)(I_{n\times n}-P)'=(I_{n\times n}-P)$ since $(I_{n\times n}-P)$ is idempotent.

Now, use this as $V$ in your procedure and that should give you the correct result.

Daniel Ordoñez
  • 844