Least Squares (LS) fitting

Question

I Have a Least Squares problem. I have a very large data set $Z$, define as follows:

$Z = \begin{bmatrix} z_0 \\ z_1 \\ \vdots \\ z_{N} \end{bmatrix}$

First, I want to fit it into the following polynomial: $z_k = a + cx_{k}^2$. after that, the residual error can be calculated: $J_{1} = \tilde{Z}^{T}\tilde{Z}$

then, i'm trying to fit it into $z_k = a + bx_{k} + cx_{k}^2$, this will result in diffrent residual - $J_{2}$.

MY question: is $J_{2} \leq J_{1}$. It's seemed like that since b can always be taken as $0$. is there any proof for that? proof for general case when increasing the model will result in a smaller residual?

Thank you all.

Obviously more "degrees of freedom" in your model results in smaller residual! If you try to fit a model in $\mathcal P_n$ polynomials of degree at most $n$, then $\mathcal P_k \subset \mathcal P_n$ for $k\leq n$. — Andrew, Jun 16 '23 at 04:19
@AndrewZhang I don't think that more degrees of freedom guarantees smaller residual. It's only if the space with more degrees of freedom is a superset of the space with smaller degree of freedom. — Acccumulation, Jun 16 '23 at 04:24
@Acccumulation Yes you are right, I should have made this clear. — Andrew, Jun 16 '23 at 04:26
just to make things clear. Is it true for any set of data $Z$? — Eden Meirovich, Jun 16 '23 at 05:02

score 1 · Answer 1 · answered Jun 16 '23 at 04:35

If $A \subset B$, then $\min_{x \in B} f(x) \le \min_{x \in A} f(x) $. This is because if $\min_{x \in A} f(x) = a$, then $\exists x \in A : f(x) = a$. Since $A \subset B, x \in A \rightarrow x \in B$. Thus, $\exists x \in B :f(x) =a$. The minimum is less than or equal to all other values, by the definition of "minimum", so $\min_{x \in B} f(x) \le a$.

One could still ask whether the LS gives the minimum (proof by nomenclature is a bit of a fallacy). There is also the issue of whether "least" here is a minimum. In some contexts, it's actually the infimum. But there is a similar, albeit more complicated, proof that this holds for infima in addition to minima, and LS does have indeed have a minimum anyway.

Least Squares (LS) fitting

1 Answers1