I am attempting to better understand matrix calculus. Given a vector $x$, I understand that: $\frac{\mathrm{d} x^t}{\mathrm{d} x}= I$ but struggle to think about $\frac{\mathrm{d} x}{\mathrm{d} x}$. This website says it is also the identity matrix, but I struggle to understand why. Would taking the derivative of a column vector wrt a column vector produce a tensor? Any intuition is greatly appreciated!
Asked
Active
Viewed 363 times
1 Answers
1
If you use the definition for vector-on-vector differentiation used on Wikipedia here, then the derivative of a column vector with respect to another column vector is indeed a matrix. To provide some intuition, consider $x'=f(x)$ as defining a coordinate transformation. (That is, $x_1'=f_1(x), x_2'=f_2(x)$, etc.) Then $dx'/dx$ is just the Jacobian matrix for this transformation, and it shouldn't be surprising that the identity transformation $f(x)=x$ has the identity matrix as its Jacobian.
Another route is to think in terms of linearization: If $y=f(x)$, then $$f(x+\epsilon h)=f(x)+ϵf′(x)h+\mathcal{O}(\epsilon^2)=y+ϵ (dy/dx) h+\mathcal{O}(\epsilon^2)$$ So the linearization of $f(x)$ only works if $(dy/dx)h$ is another column vector, i.e., $dy/dx$ is a matrix. (If you want a headache, try thinking about the $\epsilon^2$ terms!)
Semiclassical
- 15,842
-
So we can. simply say $\frac{\mathrm{d} x^t}{\mathrm{d} x}=\frac{\mathrm{d} x}{\mathrm{d} x}$ – Bepop Feb 10 '23 at 19:53
-
This does seem to be the case, but it may depend on the exact conventions used. As a point of support, though, Gentle's text on Matrix Algebra takes $\partial f/\partial x=\partial f/\partial x^t$ as true by convention (source). But in Gentle's case one then has between $\partial f/\partial x^t=(\partial f/\partial x)^t =(\partial f^t/\partial x)^t$, so the situation is a bit subtle. – Semiclassical Feb 10 '23 at 21:01