0

What is

$$\frac{\partial Ds}{\partial{D}}$$

where $D \in R^{m \times n}$ and $s \in R^n$?

I'm using the "denominator layout", so I know that this has shape $m \times n \times m$.

My guess is

$$S + \frac{\partial s}{\partial D}D^T$$

where $S_{ijk} = \delta_{ik}s_j$.

Neil G
  • 2,479
  • The shape of the gradient is actually $(m\times m\times n)$ – greg Nov 26 '19 at 04:52
  • @greg I don't see why? The denominator has shape $m \times n$, so the first two dimensions should match? But it looks like the last dimension should be $m$, which I'll correct now. – Neil G Nov 26 '19 at 05:08
  • @greg I see, you're using the "numerator layout". – Neil G Nov 26 '19 at 14:39

1 Answers1

2

You could approach this with index notation. Renaming the matrix D to A:

$$ \eqalign{ y &= As\\ y_i &= A_{ij}s_j \\ dy_i &= dA_{ij}s_j + A_{ij} ds_j\\ \frac{\partial y_i}{\partial A_{mn}} &= \bigg(\frac{\partial A_{ij}}{\partial A_{mn}}\bigg)\,s_j + A_{ij}\bigg(\frac{\partial s_j}{\partial A_{mn}}\bigg) \\ &= \big(\delta_{im}\delta_{jn}\big)\,s_j + A_{ij} \mathcal{G}_{jmn} \\ &= \delta_{im}s_n + A_{ij} \mathcal{G}_{jmn}\\ &=\mathcal{S} +A \cdot\mathcal{G} } $$

where $\mathcal{S}$ and $\mathcal{G}$ are third-order tensors and the resulting gradient is of shape $m \times m \times n$ (depending on your layout).

edited: now considering the possible dependence of s and A, introducing the tensor $\mathcal{G}$.

Traws
  • 541