Linear regression least square method

Question

Right now, I'm studying linear regression using the least squares method. So, if $f(x)=ax+b$, I have to find $$\min\sum(y-ax-b)^2.$$ But why does this minimum exist, and is there only one minimum or multiple of them?

Answer 1

We are minimizing $$Q(a,b) = \sum (y_i -a x_i-b)^2.$$

Since $Q$ is a differentiable function in $\mathbb{R}^2$, any minimiser must be a stationary point, hence a solution of $$ \frac{\partial Q}{\partial a} = \sum 2 (-x_i)(y_i-ax_i-b) = 0 $$

$$ \frac{\partial Q}{\partial b} = \sum 2 (-1)(y_i-ax_i-b) = 0 $$

i.e.

$$ \begin{cases} 1\cdot b + \bar x \cdot a =& \bar y\\ \bar x \cdot b + \overline{x^2} \cdot a =& \overline{xy} \end{cases} $$

This is the so-called normal system. When $\overline{x^2}-\bar x^2\ne 0$, that is, when the variance is not zero, the system has one and only one solution. This solution must in fact be a global minimum, due to the convexity of $Q$. As it was pointed out, the Hessian matrix, given by $$ \begin{pmatrix} 2n & \sum x_i^2\\ 2 \sum x_i & 2\sum x_i^2, \end{pmatrix} $$

is positive definite when the variance of $x$ is positive (just compute the minors' determinants).

Finally, you can try to figure out what happens when the variance is zero (all x's are the same).