What's the difference?

Question

In my work we fit a parabola to some data in order to determine three parameters. I recently talked to someone who pointed out that the ISO standard related to the fit equation had changed. The claim is that there is some subtlety with the fit which requires using the new form. I would like a second opinion.

The older standard lists the equation as

d^2 = a + b*z + c*z^2

The fit of the data to this curve can be accomplished using a linear least squares method.

The new form of the equation is

d = SQRT(a + b*z + c*z^2)

The fit of the data to this equation is done using the Levenberg-Marquardt algorithm.

My question is, what is the difference? During an early version of our software we estimated the parameters in the first equation using the linear least squares method followed by fine tuning using the Levenberg-Marquardt method. Seems to me that the results should be identical if both fits are performed using the LM method. Perhaps the “subtlety” is comparing a linear least squares method result in the first form to an LM result in the second form.

Do you have any insight on this question?

Answer 1

Aside from the issue of branches of the square root (which I'll neglect by assuming that everything is nice and positive / real where it should be to avoid imaginary numbers), there is no difference in the equations. However, if you used the Levenberg-Marquardt algorithm by directly using the first equation, and then directly using the second equation, you will get slightly different results for the coefficients $(a,b,c)$. They are clearly different problems; one minimizes the squared error between the square of your data and $d^2$, and the other minimizes the error between your data and $d$. In other words, the error functional you're trying to minimize is explicitly different, depending on how you formulate the problem.

Error metric 1: $$ e(a,b,c) = \sum_{i}\left|d_i^2-(a+bz_i+cz_i^2)\right|^2 $$ Error metric 2: $$ e(a,b,c) = \sum_{i}\left|d_i-\sqrt{a+bz_i+cz_i^2}\right|^2 $$

These are different functions; minimizing one vs. the other can, in general, give you different values for $a,b,c$.

For the linear problem (Error metric 1 above), either the LM or ordinary least-squares (OLS) algorithms work, and will give the same result. For the second problem (Error metric 2), OLS can no longer be used because the problem is nonlinear. Solving problem 2 with LM gives a different answer than solving problem 1 by either (equivalent) approach.

If problem 2 is in the ISO standard, then it is perfectly acceptable to solve the closely related problem 1, and use that solution as an initial guess to problem 2. Solving a linear approximation of a problem and then using that as an initial guess in the fully-nonlinear problem is a very good approach that minimizes the time spent iterating (hunting for a solution) in the nonlinear solver.