Let's use the following defininition of the derivative (Hubbard, 5th ed.)
Let $U \subset \mathbb R^n$ be open and let $f:U \to R^m$; let $a \in U$, $ h \in \mathbb R^n$
If there exists a linear transformation $[Df(a)]:\mathbb R^n \to R^m$ such that
$$\lim_{h \to [0]} {( f(a+h) - f(a) - [Df(a)]h ) \over {|h|}} =[0]$$
then $f$ is differentiable at $a$ and $[Df(a)]$ is unique derivative of $f$ at $a$.
Now, let's solve the following task:
let $ A = \begin{bmatrix} a & b \\\\ c & d \end{bmatrix} $. Compute the derivative for the function $f(A) = A^{-1}$
Remark: I'll be using 2x2 matrix -> 4x1 vector transformation by traversing top left, top right, bottom left, bottom right.
If we use direct definition of the derivative, and come up with the $[Df(a)]H = -A^{-1}HA^{-1}$, where $H = \begin{bmatrix} h_1 & h_2 \\\\ h_3 & h_4 \end{bmatrix} \to [0]$
But there derivative times H is wired with H: $-A^{-1}HA^{-1}$. Moreover, this derivative times H is a $2 \times 2$ matrix, so it has 4 entries in total: $$ {1 \over (ad-bc)^2} \begin{bmatrix} -bch_4 +bdh_3 +cdh_2 -d^2 h_1 & abh_4-adh_2-b^2 h_3+bdh_1 \\\\ ach_4-adh_3-c^2 h_2+cdh_1 & -a^2 h_4+abh_3+ach_2-bch_1 \end{bmatrix} $$
Now, another approach giving pure derivative by computing Jacobian. If we interpret $A$ as vector of 4 entries, then $A^{-1}$ is also vector of 4 entries. So $$[Df(A)] = -{1 \over (ad-bc)^2} \begin{bmatrix} d^2 & -cd & -db & bc \\\\ -bd & ad & b^2 & -ab \\\\ -dc & c^2 & ad & -ac \\\\ bc & -ac & -ab & a^2 \end{bmatrix}$$
What I see is: if we factor out "-" from the first approach matrix, we'll get top left element of first approach matrix coeffs equal to first row, top right - to second row, bottom left - to 3rd, and bottom right - to 4th. So we factor out "-", then each item of resulting first-approach matrix is dot product between some vector and H:
$$ -{1 \over (ad-bc)^2} \begin{bmatrix} d^2 \\\\ -cd \\\\ -bd \\\\ bc \\\\ -bd \\\\ ad \\\\ b^2 \\\\ -ab \\\\ -cd \\\\ c^2 \\\\ ad \\\\ -ac \\\\ bc \\\\ -ac \\\\ -ab \\\\ a^2 \end{bmatrix} \bullet \begin{bmatrix} h_1 \\\\ h_2 \\\\ h_3 \\\\ h_4 \\\\ h_1 \\\\ h_2 \\\\ h_3 \\\\ h_4 \\\\ h_1 \\\\ h_2 \\\\ h_3 \\\\ h_4 \\\\ h_1 \\\\ h_2 \\\\ h_3 \\\\ h_4 \end{bmatrix} $$
Now we rearrange first vector into 4x4 matrix and we're done.
Now, my questions are:
- Even though I matched direct and Jacobian approaches, I just guessed. As I calculated Jacobian as well and I knew what I had to match and how to get from direct to Jacobian. But what If I didn't have Jacobian? Would my way of factoring out H work?(I think it will, as D is linear, but just to make sure)
- while meaning of $[Df(a)]$ is clear to me, what is meaning or intuitive interpretation of $[Df(a)]H$ (to me, this meaning now is given any arbitrary H it will give good enough approximation to $f(A+H) - f(A)$. But as $H \to [0]$, lim of $[Df(a)]H = [0]$ which makes no sense to me...
