Chain and product rule for Hadamard product differentiation

Question

(Asked a similar question before but deleted to add further detail)

Similar to this question and a related to this question, how can I apply the chain and product rule to find the Jacobian of $$ f_1(x) = (g(x) \odot A) \, h(x) \tag{$*$} $$ and $$ f_2(x) = (g(x) \odot A \odot g(x))\, v \tag{$**$} $$ for fixed vector $v \in \mathbb{R}^n$, variable $x \in \mathbb{R}^n$, $g: \mathbb{R}^n \to \mathbb{R}^n$, $h: \mathbb{R}^n \to \mathbb{R}^n$, and $A \in \mathbb{R}^{n \times n}$? I'm having trouble computing the total derivative.

For an example of ($*$), I am trying to compute the total derivative for $f(x) = (-\sin(x) \odot M)\cos(x)$ with $y := -\sin(x)$, $z := \cos(x)$, and $w := Mz$ as: \begin{align} {\rm d} f(x) &= {\rm d}\{ (-\sin(x) \odot M)\cos(x) \} \\ &= {\rm d} \{(y \odot M)z\} = {\rm d} \{ y \odot (Mz) \} = {\rm d} \{y \odot w \} \\ &= y \odot {\rm d}w + {\rm d}y \odot w \\ &= y \odot (M J_{z} \,{\rm d}x) + J_y \, {\rm d}x \odot Mz \tag{$***$} \end{align} and Jacobian $$ \frac{{\rm d} f(x)}{{\rm d}x} = y \odot (M J_{z}) + J_y \odot Mz $$ but think I am messing something up. Is ($***$) correct?

Answer 1

To evaluate a function on a vector argument, as in your example, it must be applied element-wise.

The the differential of such a function is expressed, not in terms of the usual Jacobian, but as an element-wise version of the ordinary (scalar) derivative, i.e. $$\eqalign{ &y = \sin(x),\quad &z = \cos(x),\quad &w = Mz \cr &dy = \cos(x)\odot dx,\quad &dz = -\sin(x)\odot dx,\quad &dw = M\,dz \cr &dy = z\odot dx,\quad &dz = -y\odot dx \cr }$$ where ($\odot$) denotes the elementwise/Hadamard product.

Of course, diagonal matrices can be used to eliminate the Hadamard products. $$\eqalign{ &Y = {\rm Diag}(y),\quad &Z = {\rm Diag}(z),\quad &W = {\rm Diag}(w) \cr &dy = Z\,dx,\quad &dz = -Y\,dx,\quad &dw = -MY\,dx \cr }$$ Now it becomes easy to calculate the differential and gradient of the function. $$\eqalign{ f &= -y\odot w \;= -w\odot y \cr df &= -y\odot dw - w\odot dy \cr &= YMY\,dx - WZ\,dx \cr \frac{\partial f}{\partial x} &= YMY - WZ \cr }$$