Differentiate $()=^{}$ with respect to $A$

Question

Given column vector $x$ and matrix $A$, how can we take the derivative. of $x^{T}Ax$ with respect to $A$?

Answer 1

Let us define the Frobenius product by a colon and use it's cyclic property \begin{align} {\rm Tr}\left( A^T B C \right) &:= A: BC \\ &= AC^T: B \end{align}

So, \begin{align} f(x) = x^T A x \equiv x: Ax. \end{align}

Now, we can use differentials and then obtain gradient. \begin{align} df &= x: dAx \\ &= xx^T:dA \end{align}

The gradient is \begin{align} \frac{\partial f}{\partial A} = xx^T. \end{align}

Answer 2

$\frac{df}{dA}$ is going to be a matrix. The $(j,k)$-entry of $\frac{df}{dA}$ will be $\frac{df}{da_{jk}}$, where $a_{jk}$ is the $(j,k)$-entry of $A$. It follows $\frac{df}{da_{jk}}=x_jx_k$. This is because $f(A)=x^TAx=\sum_{j=1}^n \sum_{k=1}^n a_{jk}x_jx_k$. Therefore $\frac{df}{dA} = [x_jx_k]_{jk}$.