2

I have to compute the following expression:

$$ \frac{ \mathrm{d} (\mathbf{x}- \mathbf{\mu})^T \Sigma^{-1} (\mathbf{x}- \mathbf{\mu})}{\mathrm{d} \mathbf{\mu}} $$

where $\mathbf{x}$ and $\mathbf{\mu}$ are a column vectors, $\Sigma^{-1}$ is a matrix.

I tried to do it component-wise and decomposing the matrix product in sum of products:

$$ \frac{ \mathrm{d} \sum_k (\sum_j (\mathbf{x}_j - \mathbf{\mu}_j) {\Sigma^{-1}}_{j})_k (\mathbf{x}_k - \mathbf{\mu}_k)}{\mathrm{d} \mathbf{\mu}} $$

and then selecting only one component of $\mathbf{\mu}$:

$$ \frac{ \mathrm{d} \sum_k (\sum_j (\mathbf{x}_j - \mathbf{\mu}_j) {\Sigma^{-1}}_{j})_k (\mathbf{x}_k - \mathbf{\mu}_k)}{\mathrm{d} \mathbf{\mu}_k} = \mathbf{\mu}_k (\sum_j (\mathbf{x}_j - \mathbf{\mu}_j) {\Sigma^{-1}}_{j})_k - (\mathbf{x}_k - \mathbf{\mu}_k) {\mathbf{\mu}_k \Sigma^{-1}}_k $$

What is the best way to compute this derivation? Could you show the passages?

Sam
  • 357

1 Answers1

1

I would expand it first:

$$ (\mathbf{x}- \mathbf{\mu})^T \Sigma^{-1} (\mathbf{x}- \mathbf{\mu})= \mathbf{x}^T\Sigma^{-1} \mathbf{x}- 2\mathbf{x}^T\Sigma^{-1}\mathbf{\mu}^T + \mathbf{\mu}^T\Sigma^{-1} \mathbf{\mu} $$

Then you'll have:

$$\frac{d (\mathbf{x}- \mathbf{\mu})^T \Sigma^{-1} (\mathbf{x}- \mathbf{\mu})}{d\mu}= \frac{d\mathbf{x}^T\Sigma^{-1} \mathbf{x}}{d\mu}- 2\frac{d\mathbf{x}^T\Sigma^{-1}\mathbf{\mu}^T}{d\mu} + \frac{d\mathbf{\mu}^T\Sigma^{-1} \mathbf{\mu}}{d\mu} $$

$$= 0 - 2\Sigma^{-T}\mathbf{x}+ (\Sigma^{-T}+\Sigma^{-1}) \mathbf{\mu} $$

It is just a few steps of matrix algebra, for more detailed steps take a look at the matrix cookbook: https://www.ics.uci.edu/~welling/teaching/KernelsICS273B/MatrixCookBook.pdf

Alt
  • 2,592
  • please add the d operators in the nominators when expanding – Veliko May 22 '17 at 16:09
  • Are you sure about your answer? I am trying to compute the Maximum Likelihood estimator for parameter $\mu$ of the multivariate gaussian. I should end up with $\frac{1}{N}\sum_i \mathbf{x}_i = \mathbf{\mu}$ but actually with your solution I end up with $\frac{1}{N}\sum_i \mathbf{x}_i = \frac{1}{2} \Sigma^T (\Sigma^{-T} + \Sigma^{-1})\mathbf{\mu}$. – Sam May 22 '17 at 16:42
  • The Fact that $\Sigma$ is a covariance matrix makes it symmetric. Therefore, $(\Sigma^{-T}+\Sigma^{-1}) = 2\Sigma^{-1} $, which is consistent with the solution that you expect. Take a look at my answer to this question: https://math.stackexchange.com/questions/687992/mle-of-bivariate-normal-distribution/688030#688030 – Alt May 22 '17 at 18:12