1

Let $C(Z_1)$ and $C(Z_2)$ be the auto-covariance matrices of $Z_1$ and $Z_2$ respectively.

For Matrix KL divergence, $\DeclareMathOperator{\tr}{\operatorname{tr}} MKL(P \Vert Q) = \tr(P \log P - P \log Q - P + Q)$

In my use case, I am computing $MKL(I \Vert C(Z_1))$ and $MKL(C(Z_1) \Vert C(Z_2))$.

Naturally, it is desirable to rewrite $MKL(P||Q)$ using $\log \det$ which is more stable to compute than the matrix logarithm.

For $MKL(I||C(Z_1))$, we can easily use the identity $\tr(\log A) = \log \det A$:

\begin{align} MKL(I||C(Z_1)) &= \tr(I \log I - I \log C(Z_1) - I + C(Z_1)) \\ &= \tr(-\log C(Z_1) -I + C(Z_1)) \\ &\propto -\log \det C(Z_1) \end{align}

I don’t believe such an easy solution exists for $MKL(C(Z_1)||C(Z_2))$.

In my working I somehow stumbled upon something like $\log \det C(Z_1) - \log \det C(Z_1, Z_2)$ under a slightly different formulation… - maybe this is a useful hint to someone who knows?

Does there actually exist a reformulation of $\tr(A \log X)$ or even $\tr(X \log X)$?

Richie Bendall
  • 163
  • For KL Divergence, the logarithms should be evaluated element-wise, whereas Jacobi's formula for $\operatorname{tr}(\log A)$ requires that the function is evaluated in the matrix-wise sense. – greg Dec 28 '23 at 10:31
  • Indeed, with Matrix KL Divergence $\operatorname{MKL}(P \Vert Q) = \operatorname{tr}(P \log P - P \log Q - P + Q)$, $\log$ is the matrix logarithm which means we use matrix operations all the way through. – Richie Bendall Dec 28 '23 at 11:18
  • If $X$ can be diagonalized $X= O D O^{-1}$, then $\log X = O \log(D) O^{-1}$, hence $X \log X = O D \log(D) O^{-1}$, hence $tr(X \log X) = tr( D \log(D)) = \sum_i d_i \log d_i = \sum_i \log d_i^{d_i}$. This suggests that there shouldn't be an 'easy' reformulation of the form $\log \det ...$ (of course if $X^X:= O D^D O^{-1}$, then you could say $\log \det X^X$ but I guess this does not help with practical implementation) – a_student Dec 28 '23 at 12:08

0 Answers0