0

I am quite new to the field of optimization, but have one question:

I am using python and scipy to optimize my objective function (quite complex so I dont want to write it in here). I know the analytical gradient and hessian, so all optimizers can work with analytical information. All optimizers work fine, BUT the optimizers Newton-CG, TNC, BFGS, L-BFGS-B, CG have more or less the same cost after convergence. But Levenberg-Marquardt dont. Example of more or less: Cost BFGS: 2.017036885832916 Cost TNC: 2.0170368865962947

But Levenberg Marquadt: Cost 2.0419251525955135, Tweaking the tolerance parameters (termination by change of cost function etc.) does not influence it very much, so it stays almost the same. What is the reason behind it? I know, each solver has different properties and without knowing the function no one can give an explanation, so I better ask: What COULD be the reason?

horsti
  • 7
  • Are you solving a convex problem? Is it your own implementation of Levenberg-Marquardt? – littleO Nov 17 '20 at 08:36
  • It is not my own implementation (I am using scipy). All algorithms have the same starting point. In general the function is not convex. As far as I saw it in a plot, the starting point is in the convex area of the global minimum. – horsti Nov 17 '20 at 08:40
  • Since it's a non-convex problem you could just be getting stuck in different local minimums. What do you observe if you repeat the experiment several times, using a randomly initialized starting value each time for each method? – littleO Nov 17 '20 at 10:13
  • Multiple times? Yes. The numbers I mentioned are means of solving around 300-400 objective functions (all of the same shape). Different initial values? No, due to the fact of multiple minima. – horsti Nov 17 '20 at 11:09
  • I think it might shed some light if you try randomizing the initial values. We need to know if Levenberg-Marquardt is always the loser or if that was just a coincidence. Is LM ever the winner? – littleO Nov 17 '20 at 11:19

0 Answers0