Differentiation of $\operatorname{tr}(A^TB)$ if $A$ is symmetric

Question

May be an easy question but I am quite confused. Differentiation of $\operatorname{tr}(A^TB)$ with respect to $A$ is $B$ and $\operatorname{tr}(AB)$ is $B^T$. What if $A^T = A$, that is $A$ symmetric, is the differentiation $B$ or $B^T$?

Answer 1

In terms of components, $ \operatorname{tr} (A^TB)= A_{ji}B_{ji}$ with an implicit summation. Similarly, $ \operatorname{tr} (AB)= A_{ij}B_{ji}$. This explains the two derivatives you already found. So what about if $A$ is symmetric?

The subtlety lies in the fact that if you differentiate with respect to $A_{ij}$ then you are imagining changing this component slightly and seeing what this does to the traces - but you can't do this if you're changing an off- diagonal element because you're meant to have $A$ being symmetric! Put another way, you're trying to compute $n^2$ derivatives as if they're all independent when you're really in a $n(n+1)/2$ dimensional space.

So you need to see what your definition of matrix differentiation is. The usual formal definition is that you construct something such that $$\lim_{H\to 0}\frac 1{\lVert H\rVert}| f(A+H) -f(A) - C^{ij}H_{ij} | =0$$ where we would normally take $H$ to be an arbitrary matrix and $C$ the derivative. But now we want to take only $H$ symmetric, and think of $C$ as a linear map from symmetric matrices to the corresponding change in $f$.

With the convention, therefore, that $C$ only ever gets contracted with symmetric matrices, you can check that $C^{ij} = B,B^T,\frac 1 2 (B+B^T)$ and so on give the same results! Contracting with symmetric matrices only depends on the symmetric part. This makes sense since the derivative was in a space of symmetric matrices which has a lower dimension than the space of all matrices so our notation should have some redundancy; the unique way of writing it would arise from writing $C= C_a S_a$ where $a=1,\ldots,n(n+1)/2$ labels the (strictly, dual) basis of symmetric matrices $S_a$. Then the $C_a$ has the correct dimension to be an actual derivative. Accordingly the last of the three expressions I have above makes most sense.

This last example also answers the question of the derivative $\frac 1 2 (\partial_{ij}+\partial_{ji})$ where the derivatives are with respect to components of $A$. This is the most natural interpretation of a symmetric derivative - change the component and the one opposite it, with the factor of half guaranteeing that the diagonal components are treated normally.

Basically, you were asking for the derivative along a direction which does not exist in the space of symmetric matrices. Hence you need to ask how you define things.

Answer 2

Let $A=A^t$; then $f(A_{11},\dots,B_{nn}):=tr(A^tB)=tr(AB)=\sum_{i=1}^n\sum_{j=1}^n A_{ij}B_{ji}$; it follows that

$\frac{\partial f}{\partial a_{ql}}=b_{lq}=(B^t)_{ql}$ for all $q,l=1,\dots,n$.

Your statement is then correct: use the formula above for differentiation.