0

I've got a silly question, but I don't see, where I'm wrong.

Given $x,y \in \mathbb{R}^D$ and $A$ is a symmetric matrix. $$ f(A) = (x-y)^T A (x-y)\\ \frac{\partial{f}}{\partial{A}} = (x-y)(x-y)^T $$ But if I expand brackets in $f(A)=x^T A x - 2x^T A y + y^T A y$, because A is symmetric. Then it's derivative $$ \frac{\partial{f}}{\partial{A}} = xx^T - 2xy^T + yy^T $$ which is not equal to the derivative in the first case in general. Why is that?

How I've calculated the derivative in the first case. $$ df = d((x-y)^T A (x-y)) = d(tr((x-y)^T A (x-y))) = tr(d((x-y)^T A (x-y))) =\\ tr((x-y)^T(dA) (x-y)) = tr((x-y)(x-y)^T(dA)) = \langle (x-y)(x-y)^T, dA \rangle $$ The derivative is $(x-y)(x-y)^T$.

Joseph Kirtman
  • 113

2 Answers2

2

You made a symmetry assumption when you expanded your expression for $f$, but then didn’t account for it in your derivative.

Recall that the derivative of a symmetric matrix should be symmetric (prove this if you don’t see it), and therefore your expression

$$\frac{\partial f}{\partial A} = \frac{1}{2}\left(\frac{\partial f}{\partial A} + \left( \frac{\partial f}{\partial A}\right)^\intercal\right).$$

Applying this to your second expression directly yields the first.

Edit: Another way to see this is that both expressions are in fact your derivative. This is because both are the same linear mapping on symmetric matrices. This makes sense because your mapping is the derivative with respect to a symmetric matrix, which means the only relevant “direction vectors” it can act on are already symmetric.

MathIsArt
  • 865
1

That's not how it works. You need to look at the ratio $$ \frac1h(f(A+hB)-f(A))=(x-y)^T B (x-y), $$ which of course gives $$ \lim_{h\to0}\frac1h(f(A+hB)-f(A))=(x-y)^T B (x-y). $$ This means that we have the directional derivative $$ D_AfB=(x-y)^T B (x-y). $$ Equivalently, the derivative $D_Af$ is identified with the linear map $$ B\mapsto (x-y)^T B (x-y), $$ which could be written $(x-y)^T (\cdot) (x-y)$ but this notation is awful.

Saying it in other words, you cannot simply look at $(x-y)^T A (x-y)$ and get rid of $A$.

John B
  • 16,854
  • Sorry, I don't get it. Why I cannot write $D_A f B = tr((x-y)(x-y)^T B)$? – Joseph Kirtman Nov 25 '23 at 00:07
  • What it is that you don't understand? And where did the trace come from? As always: the derivative is obtained taking the limit of the ratio in my answer, there is not other way (since all is finite-dimensional in your question). – John B Nov 25 '23 at 00:09
  • $(x-y)^T B (x-y)$ is a scalar, so we can add a trace on top and nothing changes. Then you end up with the expression above, which is also can be rewritten as $\langle (x-y)(x-y)^T, B \rangle_F$. As far as I'm concerned the derivative is the first argument of this Frobenius dot product. My question was, why if we start from different expressions corresponding to the same quadratic form the results would be different? – Joseph Kirtman Nov 25 '23 at 00:15
  • If I consider the function increment, as you suggest, for the expression $x^TAx - 2x^TAy + y^T A y$, then I end up with different result also. – Joseph Kirtman Nov 25 '23 at 00:18
  • You are thinking (it seems) that a derivative must be a number. $B$ is not the variable, it is simply a "direction". I leave it to you to make the computation in that other form of yours. We both know that the two results will be the same, so one of us made a mistake! :) – John B Nov 25 '23 at 00:22
  • I've got a function that maps symmetric matrices to scalars. The result of applying differential $D_f$ to the increment $B$, which is a matrix will be obviously a scalar. – Joseph Kirtman Nov 25 '23 at 00:26