1

Most of time Newton Method in optimization is used to find the local minimum of a function. I am wondering what would happen if we have an maximization problem. What happened to the update equation in the maximization case:

X_k+1= x_k-t*dx   OR    X_k+1=x_k+t*dx

This question was asked here

Basic Question about Newton's Method for Optimization

However there two different and opposing responses and they both some votes. I would like to know can anyone clarify this ambiguity.

user59419
  • 221

1 Answers1

3

That's because it depends a bit on which Newton method you refer to.

In the one case, it's Newton's root-finding algorithm applied to the gradient of the function: this method will find a local extremum which may be a minimum or a maximum (or a saddle point). To find which, you need further exporation (for instance, looking at second order information or at the values of the function at different extrema).

In the other case, it's the Newton gradient descent method. In this case we take steps in direction of the gradient $\nabla f$ to increase the function (to find maxima) and in the direction of the negative gradient $-\nabla f$ to decrease the function (to find minima).

Riccardo Sven Risuleo
  • 3,038
  • Do you have any reference for the Newton gradient ascent. I know gradient ascent ;it makes sense to move in the direction of gradient to increase function value. I could not find any website that uses Newton method for maximization and hence can verify the argument – user59419 May 30 '19 at 15:03
  • 1
    You can find a reference in https://en.m.wikipedia.org/wiki/Gradient_descent however, you can make the intuitive argument that if you want to find the maximum of $h(x)$, you consider $f(x)=-h(x)$. Gradient descent on $f$ will give you a minimum of $f$ and a maximum of $h$. Compute the gradient of $f$ and you find that $\nabla f= -\nabla h$ which effectively proves that gradient descento on the negative gradient drives you towards a maximum. – Riccardo Sven Risuleo May 30 '19 at 16:24
  • But your both examples refer to the same algorithm. The link under "Newton gradient descent" is exactly the method that is looking for the roots of the gradient. It is the same method that is in the first link, but its multivariate version... – John Jan 07 '20 at 23:24