I am investigating the maximum value for $h$ for a multivariate function:
$h = f(x_i)$ for $x=1 \ldots n$
and using the gradient method:
$x_i = x_0 + \lambda \nabla x_i$
with $\lambda$ a small scalar to update $x_i$. However the function is converging way too slow and I need to speed up its convergence using Newton's method with a numerical Hessian matrix.
SAS v8 Finite Differences states that for central difference approximations that:
$g_i = \frac{\partial f}{\partial x_i} = \frac{f(x+h_i e_i) - f(x-h_i e_i)}{2h_i}$
for 1st order derivatives and that:
$ \frac{\partial^2 f}{\partial x\,^2_i} = \frac{-f(x+2h_i e_i) + 16f(x+h_i e_i) - 30f(x) + 16f(x-h_i e_i) - f(x- 2h_i e_i)}{12h\,^2_i}$
and
$ \frac{\partial^2 f}{\partial x_i\partial x_j} = \frac{f(x+h_i e_i + h_j e_j) - f(x+h_i e_i - h_j e_j) - f(x -h_i e_i + h_j e_j) + f(x- h_i e_i - h_j e_j)}{4h_i h_j}$
for numerical approximation of the second derivatives.
I am not sure what $e_i$ in the first partial derivative above, as I am simply using $\triangle x = h_i e_i$, a small value compared to $x$, and finding the first partial derivatives that way.
My question is this, assuming that I set $h_i e_i = h_j e_j$ for the second derivatives, then the second one for $\frac{\partial^2 f}{\partial x_i \partial y_i}$ becomes:
$ \frac{\partial^2 f}{\partial x_i\partial x_j} = \frac{f(x+ 2h_i e_i) - 2f(x) + f(x- 2h_i e_i)}{4h^2_i}$
Is this correct? Can I safely use this an an approximation for the numerical Hessian matrix entries?
Some one in another thread Summary of Optimization Methods stated that
$H \approx J^T \, J$ where $J$ is the Jacobian. Will this work instead?
