this is a question about vector differentiation.
I had to optimize the following function: $(b-Ax)^T(b-Ax)$
$A$ is matrix, $x$ and h are column vectors
$$A(t) = A^T$$
I worked this out and computed the differential, $df$ which was:
$$h(t)A(t)Ax + x(t)A(t)Ah - b(t)Ah - h(t)A(t)b$$
My question is, how do I compute the derivative $D$ from this and get a solution of the optimal $x$?
What you are doing is called least-squares minimization; you are trying to find
$$\min_x \|Ax-b\|^2,$$
whose KKT conditions are $$A^TAx = A^Tb.$$
You derive these exactly the way you started, by differentiating the objective with respect to $x$:
$$dx^T A^TAx + x^TA^TAdx - b^TAdx - dx^TA^Tb$$
and noting that a scalar is equal to its transpose, so the above can be written as $$dx \cdot 2(A^TAx - A^Tb).$$
Setting this derivative equal to zero for all infinitesimal motions $dx$ away from the critical point gives the condition above for optimality, $$A^TAx = A^Tb.$$
If $A^TA$ is invertible then finding the minimizer $x$ is easy: $x = (A^TA)^{-1}A^Tb$. Otherwise, $A^Tb$ is always in the column space of $A^TA$, and you can find one of the (infinitely many) minimizers by e.g. using the QR decomposition of $A$.
