I was trying to get some insight into how to solve non-linear regression model problems. Unfortunately, I've never attended a lecture on statistical math.
Here is the link: In page number 4, they said, calculate the least square regression. I don't understand what they mean by that. I tried searching for it. Could you give me an insight for that.
Regression problems for one explained variable of one independent variable are generally of the form
$$f(x,y;a,b,c\cdots)\approx0$$where you want to find the "best" values of the unknown parameters $a,b,c\cdots$ for a given data set $(x_i,y_i)$.
The common procedure is to minimize the global quadratic error, computed as
$$\delta(a,b,c\cdots)=\sum_{i=1}^nf^2(x_i,y_i;a,b,c\cdots).$$
(The square makes sure that all terms are positive and do not compensate each other.)
This is called the least-squares formulation. It is often intractable by analytical methods, and a numerical solution is searched by the sophisticated Levenberg-Marquardt algorithm.
On the other hand, the particular case of an affine model, $y=ax+b$ is quite accessible.
$$\delta(a,b)=\sum_{i=1}^n(y_i-ax_i-b)^2.$$ The optimum is found by canceling the partial derivatives on $a$ and $b$:
$$\frac{\partial \delta(a,b)}{\partial a}\propto\sum_{i=1}^nx(y-ax-b)=\sum_{i=1}^nxy-a\sum_{i=1}^nx^2-b\sum_{i=1}^nx=0,$$ $$\frac{\partial \delta(a,b)}{\partial b}\propto\sum_{i=1}^n(y-ax-b)=\sum_{i=1}^ny-a\sum_{i=1}^nx-b\sum_{i=1}^n1=0.$$
and this forms an easy $2\times2$ system of linear equations.
From this useful tool called linear regression, other nonlinear problems can be solved if they can be put in the form
$$y=g(af(x)+b),$$i.e.$$g^{-1}(y)=af(x)+b.$$
In this case, instead of performing the linear regression on $(x,y)$, you do it on $(f(x),g^{-1}(y))$.
For example, the power law $y=bx^a$ becomes $\ln(y)=a\ln(x)+\ln(b)$ by taking the logarithm.
When you perform a linear regression, you assume that your model is $Y = a + b X$ and your search for parameters $a$ and $b$ which minimize the sum of squares of the residuals (predicted $Y$ - given $Y$).
The model is said linear because the derivative of $Y$ with respect to any of the parameters does not involve any parameter.
Now, suppose you want to fit a model which is $Y = e^{a + b X}$. From what I told you before, this model is nonlinear with respect to its parameters since the derivative of $Y$ with respect to $a$ involves $b$ and the derivative of $Y$ with respect to $b$ involves both $a$ and $b$. For the nonlinear models, the key problem is to be able to start with reasonable guesses.
In the case of the model $Y = e^{a + b X}$, you can notice that, if we take the loarithms of both sides and define $Z = \log(Y)$, we end with a linear model $Z = a + b X$. So, now, we can apply standard linear regression to obtain estimates of parameters $a$ and $b$.
So let us take the data on page 6 and use what has been described above. The linear regression leads to $a = 10.5793$ and $b = -0.2184$. These values will now be used for the nonlinear regression; this leads to $a = 10.6047$ and $b = -0.2244$ which are "close" to the initial values but different.
The reason the parameters are different is related to the fact that in the linear model we minimized the sum of squares based on the $\log(Y)$'s while in the nonlinear model we minimized the sum of squares based on the $Y$'s.
Concerning more mathematical aspects, I suggest you have a look at
http://en.wikipedia.org/wiki/Non-linear_least_squares
http://en.wikipedia.org/wiki/Non-linear_least_squares#Solution
http://en.wikipedia.org/wiki/Curve_fitting
Please let me know if you need more clarifications.
