1

I'd like to know how to choose a good optimization method for one problem. I know that it depends of the search space (mainly smoothness and modality).

The issue is that you need to explore and observe this search space. That is to say, you need to generate all combinations of results (brute force) to plot it. It makes the algorithm/method useless, beacause once your search space is generated, you already know the minimum or maximum of the function, that's what you are looking for.

How to know which method to use without brute forcing ?

mathreadler
  • 25,824
Olstrom
  • 11
  • Understanding how Mathematica chooses the Method in NMiminize (global) or FindMinimum (local) could be helpful, maybe (I have no clue on the answer). – anderstood Sep 20 '17 at 01:03

2 Answers2

1

Several factors come into play and may influence the choice of the method you want to use. To cite a few:

  1. The computation time of the Jacobian. If you use descent methods and there are a lot of variables, the way you compute the Jacobian will be essential.

  2. The computational time of the cost function. If it is very quick to compute (compared to your available time), you might think of metaheuristics, which are a completely different class of algorithms.

  3. The number of variables

  4. The shape of the cost function: with a lot of local minima, you might get stuck and miss the global minimum when using descent methods

Of course those two points are coupled, and there is no systematic solution methods (it's a bit like cooking...).

anderstood
  • 3,504
  • Thanks for your answer. My original question was more about : how do you find out the shape of this cost function ? You should compute all the possibilties which makes the optimization method useless. – Olstrom Sep 19 '17 at 14:10
  • This issue is exlained here : http://www.redcedartech.com/pdfs/Select_Optimization_Method.pdf – Olstrom Sep 19 '17 at 14:12
0

You can't. Mathematicians hate brute force and only do it when there's no other choice. There's no general theorem or criterion that will tell you the best algorithm to use.

Stella Biderman
  • 31,155
  • Thanks for your answer @Stella.

    You check several results with several method, like gradient method, then newton, correlation based method, genetic, etc ... and you just choose the best ?

     – Olstrom Sep 19 '17 at 14:00
  • @Olstrom Instead of trying the methods, you can try visualizing the cost function (potentially by fixing some parameters). If you see a lot of local minima, you might prefer genetic to gradient method, for instance. – anderstood Sep 20 '17 at 00:58