Derivatives of second order

Question

Consider a real function $f$ of one variable. Suppose the second order derivative exists. To find the second order derivative of $f$, I usually derivate $f$ two times. I start with $f$, and derivate to get the function $\frac{d}{dx} f(x)$ of one variable, which I then derivate again. I would succinctly describe this by writing

$$\frac{d}{dx} \frac{df}{dx}.$$

But when I consider a real multivariable function which maps elements into $\mathbb{R}^k$, with $k \geq 1$, I do not always know how to find the second order derivative. In the case $k=1$ we can consider the derivative of some function $g$ as the gradient $\nabla g$ of $g$. Then the second order derivative of $g$ would be the hessian matrix $H(g)$ of $g$.

But when $k \geq 2$, for example $h(x, y)=(x^2, y^2)$, I do not find it easy to compute the second order derivative, and our coursebook does not discuss this topic.

I know that

$$[h'(x, y)]=\begin{pmatrix} 2x & 0 \\ 0 & 2y \end{pmatrix}.$$

The difference between finding the derivative of $f$ and $g$ or $h$ is that when derivating $f$ we get a function of the same structure, we still map elements from $\mathbb{R}$ into $\mathbb{R}$, but when derivating $g$ or $h$ the structure changes and instead of, for example, mapping elements from $\mathbb{R}^2$ into $\mathbb{R}^2$, we map elements from $\mathbb{R}^2$ into $\mathbb{R}^{2 \times 2}$ (the set of 2 by 2 matrices with real elements). To me, it seems that the problem of finding the second order derivative of $g$ is to derivate a matrix in which all elements are real functions.

I would be grateful if you could, for example, direct me to any material that would help me solve this general problem.

Thanks.

Answer 1

If we have some function $f \ \colon \mathbb R^n \to \mathbb R ^m$ then the derivative of $f(\underline{x})$ (if it exists) is an $m \times n$ matrix. Since this derivative varies as we change the input, we can describe it as a function $f' \ \colon \mathbb R ^n \to \mathbb R ^{m \times n}$ which assigns to each input its derivative matrix. Note that while the matrix will be a linear operator, its entries will not depend linearly on $\underline{x}$. For example the matrix $ \left( \begin{array}{ccc} 3x^2 & 0& 0\\ 0& 3y^2& 0\\ 0 & 0 & 3z^2 \end{array} \right) $ Is the derivative of the function $f(x,y,z) = (x^3, y^3, z^3)$. The entries are clearly not linear functions.

The derivative of $f'(\underline x)$ is the derivative of a function $\mathbb R ^n \to \mathbb R ^{m \times n}$, so it will be a $(m \times n) \times n$ matrix. This is easier to imagine as a "cube" of mixed partial derivatives. In our previous example the first derivative could be thought of as a function that assigns to each pair $(i,j)$ the entry $\displaystyle \frac{\partial f_j}{\partial x^i}$ of the corresponding matrix. The second derivative will assign to each pair $(i,j,k)$ the entry $\displaystyle \frac{\partial}{\partial x^i} \frac{\partial f_k}{\partial x^j}$ in the "cube". The "cube" is called a second rank tensor, and a matrix is called a first rank tensor. A $n ^{th}$ rank tensor is a linear operator which inputs $n$ vectors and outputs one.

Ultimately, computing $f^{(n)}(\underline{x})$ amounts to computing each $\displaystyle \frac{\partial}{\partial x^{i_1}} \frac{\partial}{\partial x^{i_2}} \ldots \frac{\partial f_{i_{n+1}}}{\partial x^n}$. Depending on your function several of these can be done at once . It is rarely helpful or useful to try and illustrate all of the mixed partial derivatives as some form of "hypercube" however.