1

Suppose we want to find the best coefficients $a$ and $b$ that fits the data we have according to a model of the form $$ y = a t e^{bt} \text{ or } y = a e^{bt} \text{ or } y = a \left( \frac{x}{b+x} \right) $$ for example. Then I know we can linearize these non-linear least squares problems and find the linear regression of the transformed problem. My question is : will the values of $a$ and $b$ found by solving the linearized problem actually be the optimal parameters for the original non-linear least squares problem ?

I searched a lot on the Internet for information about this and it seems like in some cases we will get the exact values and in other not because the errors according to the linearized problem will behave differently compared to the original non-linear problem. For example computations seem to show that for the first model $y = ate^{bt}$, we get the same $b$ when solving the transformed linear model and numerically calculating the solution of the non-linear model, but $a$ differs a little. But, I do not quite understand. Any help will be very appreciated.

Guest
  • 4,158
  • No, the linearized solution will approximate (but not equal) the actual, nonlinear solution. But you can improve the approximation by successive linearization. You may want to read up on Newton's method and its variants, in particular Gauss-Newton. – user7530 Oct 22 '13 at 01:04
  • Ok thank you I will read about it. But just to make sure : when you say that the linearized solution will not equal the actual nonlinear solution, that is always true ? That is, we can never get the actual nonlinear solution by linearizing the problem ? – Guest Oct 22 '13 at 01:08
  • That depends on the exact formulation of the fitting function... but in general, no, you cannot find the exact solution. But you can come within a very small error of the exact minimizer, if you use numerical methods to iteratively linearize. – user7530 Oct 22 '13 at 02:24

2 Answers2

1

The problem with nonlinear regression is the need of "good" estimates of the model parameters. When linearization is feasible, it is a very good way to get them; when linearization is not feasible, the preliminary task of finding reasonable estimate is quite tedious and may require a lot of effort. Somletime, the model is linear with respect to all parameters except one : in such a case, fix this parameter, perform regression and iterate manually until you find a minimum of the obhjective function. From there, you can start. But, the parameters obtained by linearization can be very far away from those corresponding to the original problem. One of the key reasons is that we do not deal with the same objective function.
Suppose you have a model Y = Exp[a + b / T]; what you want to minimize is the sum of squares of the errors on Y. You can easily linearize this model; but, you will then minimize the sum of squares of the errors on Log(Y) which is totally different. More serious is the fact that regression must be done on what is measured; in my simple case, Y is measured, not Log(Y).

Claude Leibovici
  • 260,315
  • Thank you for sharing your insight. When you say that the parameters obtained by linearization can be very far away from those corresponding to the original problem, is that a rare situation ? I linearized the problem of minimizing the squares of the errors on $y = a t e^{bt}$ and minimizing the squares of the errors on $Y = \log(y/t)$ gave me exactly the same value of $b$ as when I numerically solved the non-linear least squares problem for $y$. Also $a$ is very close in both cases... I guess that's because of the particular data I had ? – Guest Oct 22 '13 at 04:17
  • @Guest. If the data are in very small errors, the parameters will be close to each other ... but not the same. Could you post your data ? – Claude Leibovici Oct 22 '13 at 05:06
  • Yes. I have the following points $(t, y(t))$ : $(1, 8), (2, 12.3), (3, 15.5), (4, 16.8), (5,17.1), (6, 15.8), (7, 15.2)$ and $(8, 14)$. – Guest Oct 22 '13 at 20:53
0

@Guest. Thank you for having posted your data. I did the same as you. Using the linearized model leads to a = 9.790160 and b = -0.2151368. Using the nonlinear model leads to a= 9.796928 and b =-0.2150872. They are not very different but they are not the same. As I said, they are close because the data are not in large error.

But again remember that what was measured is Y not Log(Y/t) !

I hope this help. PLease let me know how you react.

Claude Leibovici
  • 260,315