0

I have a loss function which I want to minimize. That function depends on 4 variables, which need to be tweaked to find the minimum. There is no need for finite differences, because I know the analytical function. The function is non-convex, but in the area of my initial guess it is convex.

Question: Which optimization method should I use? Quasi-Newton Methods (e.g. BFGS) or the "full" Newton method? Should I use Conjugate gradient (but I've read that it is better for larger dimensions). Thanks for your help :)

Horsti

horsti
  • 7

1 Answers1

0

It depends. Your function is non convex, ok, but do you know if it has multiple local/global minima? And if it does, do you care about finding the global optima or are you happy with a good local minimum?

Four variables may not seem much but I’ve seen extremely tough objective functions with 4 variables. If you wish to find the global optimum of your function, then a global optimization algorithm (simulated annealing, evolutionary, or deterministic ones such as MCS and SHGO) might be able to give you a robust answer. They are of course much slower than local optimization algorithms such as BFGS as they have to explore much larger regions to find a global minimum. BFGS and friends will only give you a local optimum - unless you’re already very close to the global one.

But since you have the analytical formula for your objective function, did you try and explore its properties via the gradient and hessian - potentially using a symbolic math package?

Infinity77
  • 257
  • Yeah it has multiple minima but usually my initial guess is in a convex region. So its very likely that if I found the local minima it is the global one. Therefore it is enough to seek for the local minimum. What do you mean with its properties? Do you mean to seek analytically for the minimum? If so, No I didn't, I was far to lazy to do so (the equation is quite long). What do you mean with symbolic math package? A package which is doing what I was to lazy for? Thanks for your answer :) – horsti Sep 24 '20 at 07:26
  • I am not sure what you mean when you say “my initial guess is in a convex region”. Unless your function is convex across the entire domain of definition, then you cannot say that your local minimum is the global one. What I meant by symbolic math package is a package (like SymPy for Python, or Maple or Mathematica) that allows you to manipulate your analytically defined function and calculate derivatives, hessian, zeros and so on. – Infinity77 Sep 24 '20 at 15:32