1

In a normal linear model (with intercept), show that if the residuals satisfy $e_i = a + \beta x_i$, for $i = 1\dots n$, where $x$ is a predictor in the model, then each residual is equal to zero.

I'm not really sure how to do this, I tried writing out $e_i = y_i - \hat{y}_i$; but I wasn't able to get anywhere.

mrnovice
  • 5,773

2 Answers2

1

If our linear model is given by

$$y_i=w_0+w_1x_i+e_i$$

we can substitute $e_i=\beta_0+\beta_1x_i$ to obtain $$y_i=w_0+w_1x_i+(\beta_0+\beta_1x_i)$$ $$\implies y_i=\left[w_0+ \beta_0\right]+\left[w_1+\beta_1 \right]x_i$$ $$\implies y_i=\tilde{w}_o+\tilde{w}_1x_i+\tilde{e}_i,$$

in which $\tilde{e}_i=0$. Hence, as the error is not random. The model can capture everything that is present in the error. Residual term $\tilde{e}_i\equiv 0.$

MachineLearner
  • 3,991
  • Why can we write the linear model in this way: $y_i = w_0 + w_1x_i +e_i$? Shouldn't it be $y_i = x_i^T \beta + \epsilon_i$, where $\epsilon$ is normally distributed? – mrnovice Mar 07 '19 at 23:21
  • Both expressions are the same (I just used a linear regression). The problem is that they state that the residual term has some specific form. – MachineLearner Mar 07 '19 at 23:27
  • But if our model has $p$ predictors, shouldn't there be $w_0,w_1,...,w_{p}$ in your first equation? Forgive me if I'm being stupid, but I don't understand how to go from my expression to your answer. Could you maybe add an edit showing how to do that? – mrnovice Mar 07 '19 at 23:52
  • @mrnovice : MachineLearner has shown that under the given conditions, the observation vector $Y$ must lie in the column space of $X$, so we must have $\hat{Y} = Y$ or $e = 0.$ – Arin Chaudhuri Mar 08 '19 at 03:37
1

Since your regression model has intercept, we can assume, the X matrix for the regression has the form $$ X = \begin{pmatrix} 1 & x_1^T \\ 1 & x_2^T \\ \vdots & \vdots \\ 1 & x_n^T \\ \end{pmatrix} $$

Note that the residual must be orthogonal to every vector in column space of $X$. This is because the predicted value, $\hat{Y} = P_X Y$ (where $P_X$ is the orthogonal projection matrix onto the column space of $X$), and hence the residual vector $e = Y - \hat{Y} = (I - P_X)Y $. So for any vector $c$ of appropriate dimension $$c^T e = (c^T - c^TP_X)Y.$$ Now if $c$ lies in the column space of $X$ then $$P_X c =c$$ or $$c^T = c^T P_X^T = c^T P_X \text{ since $P_X$ is symmetric}$$ and it follows for any $c$ in the column space of $X$ $$c^T e = 0.$$

Now, your condition implies $e$ itself lies in the column space of $X$ and hence must be orthogonal to itself, i.e., $e^Te = \|e\|^2 = 0$, i.e., $e = 0.$

Arin Chaudhuri
  • 6,349