0

I'm trying to understand simple linear regression. Here is a problem I'm working on, and I'm trying to understand the answers conceptually.

I want to find the least square estimators $b_1$ and $b_0$ for the following:

(a) The "no slope" model $Y_i = \beta_0 + \epsilon_i$, and

(b) The "no intercept" model $Y_i = \beta_1x_i + \epsilon_i$.

To do this, I know that I will need to minimize the function $Q = \sum_{i=1}^{n} \epsilon_i^{2}$. Thus, I have

(a) $Q = \sum_{i=1}^{n} \epsilon_i^{2} = (Y_i-\beta_0)^2$. Now minimizing this yields,

$0=\frac{\partial Q}{\partial \beta} = \sum_{i=1}^{n} -2(Y_i-b_0) \Rightarrow 0=\sum_{i=1}^{n}Y_i-nb_0 \Rightarrow b_0 = \overline{Y}$.

(b) similarly for this part, I get that $b_1 = \frac{\sum_{i=1}^{n}y_ix_i}{\sum_{i=1}^{n} x_i^{2}}$.

I understand that part (a) makes intuitive sense -- if there is no slope, then the regression line is a horizontal line with value $\beta_0$. So, even though $\beta_0$ is an unknown parameter, we can still reason that the best estimate would be whatever its value is. I'm looking for a way to understand the answer for part (b) in the same way (it seems less intuitive to me). Thanks for any explanation.

mXdX
  • 571
  • 3
  • 14

2 Answers2

2

The algebraic part: $$R(b_1)=\sum_{i=1}^n (Y_i-\hat{Y_i})^2=\sum_{i=1}^n (Y_i-b_1X_i)^2\\ R'(b_1)=2\sum_{i=1}^n(Y_i-b_1X_i)(-X_i)=-2\sum_{i=1}^n X_iY_i+2b_1\sum_{i=1}^nX_i^2=0 \Rightarrow \\ b_1=\frac{\sum_{i=1}^n X_iY_i}{\sum_{i=1}^n X_i^2}.$$ The geometric (intuitive) part:

$\hspace{1cm}$enter image description here

Note that the numerator is the sum of areas of brown rectangles $X_iY_i$ and the denominator is the sum of the areas of green rectangles $X_i^2$. And their ratio (the relative weights) determines the slope of the line. Obviously, the total green area is bigger, hence the line is flatter.

farruhota
  • 31,482
0

One answer to your question about (b) is that this is the way the math works out - nothing more, nothing less.

Since this is probably unsatisfactory, I have come up with this:

The slope for each point is $\dfrac{y_i}{x_i}=s_i$. Therefore, if the result is of the form $\dfrac{\sum u_i}{\sum v_i}$, then we "should" have $u_i = v_is_i$, so the slopes weight the data. If $s_i = s$ for all $i$, then we want $s =\dfrac{\sum u_i}{\sum v_i} =\dfrac{\sum v_is_i}{\sum v_i} =s $, but to ensure that $\sum v_i \ne 0$, the simplest way is to have $v_i = x_i^2$. Then $u_i = v_is_i =x_iy_i$.

Of course, while doing this my hands are waving so fast that I might take flight. But this is the best that I can do right now.

As I wrote at the start, it is really the fact that this is the way the math works out.

marty cohen
  • 107,799