3

I'm a beginner at matrix calculus and I've been using the Wikipedia page for my rules. I'm having some trouble with this problem that I've faced. I know the solution, but I'm wondering why my strict application of the matrix calculus "rules" doesn't seem to massage out the right answer. Here's the problem:

Find the derivative of the function below with respect to $u_i$.

$$L = \sum_{ij}(u_i \cdot v_j - R_{ij})^2$$

It seems relatively straightforward, but when I apply the rules using the "denominator method" of notation, I seem to run into a dimension mismatch:

$$ \frac{dL}{du_i} = \sum_j 2(u_i^T v_j - R_{ij})v_j = 2\sum_j u_i^T v_jv_j - R_{ij}v_j $$

Note that both $u_i$ and $v_j$ are column vectors and that, therefore, $v_jv_j$ does not make sense. The main property I'm using is:

$$\frac{dg(z)}{dx} = \frac{dg(z)}{dz} \cdot \frac{dz}{dx}$$

Which seems like simple chain rule. Am I misunderstanding something important with how this algebra works out? Thanks!

ImpGuard
  • 133

1 Answers1

0

In the formula you give $$ \frac{dg(z)}{dx} = \frac{dg(z)}{dz} \cdot \frac{dz}{dx} $$ it doesn't matter where the term $\frac{dg(z)}{dz}$ goes because it is a scalar. However, if you want to treat the RHS above as matrix multiplication where a scalar is a $1\times 1$ matrix, then the proper formula is $$ \frac{dg(z)}{dx} = \frac{dz}{dx}\frac{dg(z)}{dz}\cdot $$ Once you do this then your problem goes away: $v_ju_i^Tv_j$ can both be interpreted as $v_j$ multiplied with a scalar $u_i^Tv_j$ or as the matrix product $v_ju_i^Tv_j$. Note that this is consistent with the wiki's identity for $\frac{\partial g(u)}{\partial x}$ written under the denominator convention. See the 'vector-by-vector identities' in the wiki article.

Kim Jong Un
  • 14,794
  • 1
  • 24
  • 49