0

Clarification: the word "important" is ambiguous. Here, I use "important" to mean that we want to know the value of a quantity accurately.

For many optimisation problems, it looks as if we are more interested in finding the minimum value, rather than finding the argmin. For instance, in machine learning models, we care about how well the model can predict, and we do not care about the internal argmin inside the black box that makes this possible. For a linear programming problem, say, for an airport, we do not really care about the exact way to schedule flights to get the maximum efficiency. We just want to come up with a plan which can be completely different from the optimal one, but have similar efficiency in terms of saving time. This is due to the existence of many equally good solutions.

Note: one might argue that we do need to know a solution. Of course we do. But we don't care which solution we get, as long as it is a good solution. So it is unimportant to know the solution itself accurately; only the objective value is important.

Nevertheless, argmin is discussed a lot in literature (like parameter estimation in statistics). However, I have not seen anywhere why this is useful. Could anyone give an example application where the argmin is more important than the min?

Ma Joad
  • 7,420
  • ML: "we only care about how well the model can predict". One goal in statistics, at least in some applications, is to do inference on the estimated parameters. This is, perhaps, in contrast to ML. – Andrew Jun 19 '23 at 21:42
  • @AndrewZhang Yes, I see a lot of those in statistics. (Can one argue that actually estimation of parameters is not important?) – Ma Joad Jun 19 '23 at 21:49
  • 1
    In optimization we typically want to find parameter values that minimize some cost, or some objective function. That means we want to know the argmin. It is the optimal parameter values that we will use in our application; the minimum cost function value is incidental. For example in radiation treatment planning the optimal beam intensity values are used during treatment when we actually fire the beams of radiation. Even in deep learning it’s the weights I want to know, not the minimum average cross-entropy value. Give me the weights and then I can use them to make predictions. – littleO Jun 19 '23 at 21:54
  • I would think it certainly depends on the problem. For deep learning problems, the data is unstructured, raw. E.g. images or text. To us, the data really has no interpretation. However, if the data is meaningful (e.g. income, etc.), then classical statistical models tend to come into play. Chapter 2 of ISLR has some very nice discussion of why you want to care about the parameter estimates. – Andrew Jun 19 '23 at 21:56
  • 2
    "For a linear programming problem, say, for an airport, we do not really care about the exact way to schedule flights to get the maximum efficiency" Im not convinced this example supports your sentiment of only caring about the minimum value. It would seem that the actual plan here is certainly important. Telling the airport they can achieve some theoretical cost is useless if you do not tell them how to do it. – Andrew Jun 19 '23 at 21:59
  • @AndrewZhang But we do not care about how close our solution is to the optimal one. We merely care about how good our solution is to the optimal one. You see what I mean. – Ma Joad Jun 19 '23 at 22:13
  • @littleO I have never said that you do not want to know the weights. I have just said that you do not care about how close the value of weights you have to the best weights. – Ma Joad Jun 19 '23 at 22:15
  • Suppose you have noisy observations of a signal. You choose a Fourier basis and you can fit as many coefficients as you’d like. The more coefficients you fit, the smaller your training error is. But your estimated parameters will exhibit high variance, and thus be far from the true parameters. I guess my point is, if you can not say that your estimate will be close to the true parameter value, then how can you say generalization? – Andrew Jun 19 '23 at 22:27

1 Answers1

2

I would say the opposite. In every optimization problem you want to find the optimal solution and are not interested in the value of the functional on the minimum.

Emanuele Paolini
  • 21,447
  • I might need to explain more, but you missed my point completely. Of course, we want to know a solution in almost all cases. But it is a solution, not "the" best solution. See my edit. – Ma Joad Jun 19 '23 at 22:09
  • In general if it is possible to find an exact solution to a mathematical problem it is always better than finding an approximate solution. Of course if we are not able to find the exact solution then the approximate solution is the best we can do. – Emanuele Paolini Jun 19 '23 at 22:19