1

If I use Gauss-Newton to solve a least square optimization problem and $\mathbf{J}^H\mathbf{J}$ is constant does it imply that I will reach the solution in one iteration?

trienko
  • 407

1 Answers1

2

The answer is no. The reason for that is that the matrix $J^HJ$ is not the Hessian, but an approximation of it. Look at the following one-dimensional example: given constant $y$, solve $\min f(x)$ where $$ f(x)=\left\|\left[\matrix{y_1\\y_2}\right]-\left[\matrix{\cos(x)\\ \sin(x)}\right]\right\|^2. $$ Here $$ J=\left[\matrix{\sin(x)\\ -\cos(x)}\right],\qquad J^HJ=1 $$ but $f$ is not a quadratic function ($f''$ is not constant), so Newton does not converge in one step.

A.Γ.
  • 29,518
  • Can you say that you will get close to the true answer in one step? If you start reasonably close to the true value? Will you converge faster than the case where your $\mathbf{J}^H\mathbf{J}$ is not constant? – trienko Jul 14 '15 at 15:00
  • @trienko I do not think so. Local convergence is the property of Newton method (under certain conditions on the second and third derivatives), however, the Gauss-Newton is only an approximation, so the situation with convergence can be much worse here (unless you add some kind of line search to it). – A.Γ. Jul 14 '15 at 15:12