0

In this illustration the value of each iteration is the minimum of the 2nd derivative.

Animated Gauss-Newton

But the Wikipedia page says:

the advantage [of the Gauss–Newton algorithm] is second derivatives, which can be challenging to compute, are not required.

Question:

Is the illustration wrong? If yes what algorithm is used?

JnxF
  • 1,277
Vadi
  • 155

1 Answers1

1

In the illustration the second order approximation to the "original" curve and not the second derivative is shown.

Relevant to this question is probably the difference between Newton-Raphson and Gauss-Newton explained here.

If the goal was indeed to find the minimum of the second order function, knowing that it has only one extrema - the minimum, only the first derivative would be required, since that derivative has only one zero.

Also note that the y-axis says $\chi^2$, therefore its safe to assume that it is the minimum of the sum of squares which is approximated (at least indirect over the optimization of parameter a).

The chapter Derivation from Newton's method of the wikipedia article (which is more from the perspective of fitting and therefore an overdetermined system) explains the reason, why you do not need the second derivative after the line "The Gauss–Newton method is obtained by ignoring the second-order derivative terms".

Both the wikipedia article and the illustration are right. Also there is no conflicting information.

Community
  • 1
NeinDochOah
  • 613
  • I hope after a lot of edits this has finally reached a form, where the different pitfalls are clear. I think the problem was simply misreading/misinterpreting, however it could also be about the difference between fitting and just finding a zero. – NeinDochOah Jan 24 '16 at 02:33
  • is it correct to say the green curve (1st order approx.) would be used if it was Newton-Raphson and the blue curve (2nd order approx) represents the Gauss-Newton algorithm? 2nd quick question: in such case with only one variable, is it correct to say the Hessian is approximate by twice the squared of the first derivative ? – Vadi Jan 24 '16 at 22:14
  • @Vadi 1) it looks like Newton-Raphson, however if you look at the algorithm its works best to find zeros, take a look at this link. 2) The Hessian is the second derivative. Also you build your squared sum and then take the derivative. – NeinDochOah Jan 25 '16 at 10:12