Basically, I have an equation in the form
$$ f = \left \| A^{-1} \right \|^2 $$
I need to differentiate the above equation wrt A.
The matrix A is just a 2X2 matrix so I tried to solve it by brute force. It worked but the solution is very long. Not to mention, it is very time-consuming. I was wondering if there is a general method to solve this problem that will work for any order of matrix A.
Wikipedia tells us: $$\|M\|=\sqrt{\mathrm{tr}(A^*A)}$$ Thus \begin{align*} \|A+tX\|^2&=\mathrm{tr}(A^*A+tX^*A+tA^*X+t^2X^*X) \\ &=\mathrm{tr}(A^*A)+2t\,\mathrm{tr}(X^*A)+t^2\mathrm{tr}(X^*X) \end{align*} by linearity, conjugation invariance, and cyclic invariance. Looking for the linear term, $$D_A{(\|A\|^2)}(X)=2\mathrm{tr}(X^*A)$$ Likewise, $$D_A(A^{-1})(X)=-A^{-1}XA^{-1}$$ Thus by the chain rule, $$D_A{(\|A^{-1}\|^2)}(X)=2\mathrm{tr}((-A^{-1}XA^{-1})^*A^{-1})$$
$ \def\A{A^{-1}} \def\At{A^{-T}} \def\p{\partial} \def\L{\left}\def\R{\right}\def\LR#1{\L(#1\R)} \def\trace#1{\operatorname{Tr}\LR{#1}} \def\qiq{\quad\implies\quad} \def\grad#1#2{\frac{\p #1}{\p #2}} \def\c#1{\color{red}{#1}} \def\CLR#1{\c{\LR{#1}}} $The double-dot product provides a concise notation for the trace and the Frobenius norm $$\eqalign{ A:B &= \sum_{i=1}^m\sum_{j=1}^n A_{ij}B_{ij} \;=\; \trace{A^TB} \\ A:A &= \|A\|^2_F \\ }$$ The properties of the underlying trace function allow the terms in such a product to be rearranged in many different but equivalent ways, e.g. $$\eqalign{ A:B &= B:A \\ A:B &= A^T:B^T \\ C:\LR{AB} &= \LR{CB^T}:A \\&= \LR{A^TC}:B \\ }$$ Write your function using this notation, then calculate its differential and gradient. $$\eqalign{ f &= \A:\A \\ df &= {d\A}:\A \;+\; \A:{d\A} \\ &= 2\A:\c{d\A} \\ &= 2\A:\CLR{-\A\,dA\,\A} \\ &= -2\LR{\At\A\At}:dA \\ &= -2\LR{A^TAA^T}^{-1}:dA \\ \grad{f}{A} &= -2\LR{A^TAA^T}^{-1} \\ }$$
