Derivative of scalar function with respect to matrix with vectors involved

Question

I want to find derivative of a scalar function with respect to matrix $A$:

$ E=\|f(Ax)\|^2 $

Where $f(Ax)$ is a vector, say color of pixel at position $Ax$. How can I do that, given that I can compute derivative of $f$ with respect to its argument $\partial{f}/\partial{x}$?

I know that I can rewrite function $E$ like this

$ E=f(Ax)^Tf(Ax) $

or

$ E=tr(f(Ax)f(Ax)^T) $

Is there a way to find derivative using matrix-vector operations? E.g. without computing derivatives with respect to individual matrix elements. Is there a general analog of chain rule? Say for $g(f(Ax))$, where g is scalar and f is vector.

Yes. It's called the Matrix Cookbook. Start reading around page 8. https://www.math.uwaterloo.ca/~hwolkowi/matrixcookbook.pdf — Christopher Carl Heckman, Apr 17 '16 at 02:24
@CarlHeckman I've already studied cookbook and did a lot of matrix calculus in the past, yet I don't see pretty solution. — DikobrAz, Apr 17 '16 at 02:36
I wasn't sure by your post. And yes, it does become nasty, because you get $\delta A$ and $(\delta A)^\top$ in your answer ... — Christopher Carl Heckman, Apr 17 '16 at 02:45

score 4 · Accepted Answer · answered Apr 17 '16 at 04:53

For convenience, define the variables $$\eqalign{ y &= Ax,\,\,\,f = f(y),\,\,\, J = \frac{\partial f}{\partial y} \cr }$$ Then write the function in terms of the Frobenius (:) Inner Product and take its differential $$\eqalign{ E &= f:f \cr\cr dE &= 2f:df \cr &= 2f:J\,dy \cr &= 2f:J\,dA\,x \cr &= 2J^Tfx^T:dA \cr }$$ Since $dE = \big(\frac{\partial E}{\partial A}:dA\big),\,$ the gradient must be $$\eqalign{ \frac{\partial E}{\partial A} &= 2\,J^Tfx^T \cr }$$

Derivative of scalar function with respect to matrix with vectors involved

1 Answers1