1

Suppose variable $x$ is a N*1 vector, $A$ is a M*N matrix and $b$ is a M*1 vector.

$$ f(x) = \|e^{Ax} - b \|_2^2 $$

Does its derivative should be like following? $$ \frac{\partial f}{\partial x} = 2*(e^{Ax} - b)*A^T*e^{Ax} $$

If it is, but the dimension of its derivative doesn't equal to the dimension of $x$. How to explain that? Thanks!

Jinfeng
  • 13
  • How do you define $e^{Ax}$? – TZakrevskiy Apr 06 '15 at 19:18
  • This is simply the exponential value for each entry in vector $Ax$. – Jinfeng Apr 06 '15 at 19:19
  • I think the best explanation here is that your formula for the derivative is wrong. – TZakrevskiy Apr 06 '15 at 20:00
  • Yes, I found the solution from your hint. Since here $e^{Ax}$ is a element-wise operator, therefore the derivative is not equal to $A^T e^{Ax}$. However, it should be $A .* e^{Ax}$, which is in M*N dimension. This can be proved by definition when you express $e^{Ax}$ in matrix form. – Jinfeng Apr 06 '15 at 20:06

2 Answers2

0

I found the solution by myself eventually. Since here $e^{Ax}$ is a element-wise operator, therefore the derivative is not equal to $A^T e^{Ax}$. However, it should be $A .* e^{Ax}$, which is in M*N dimension. This can be proved by definition when you express $e^{Ax}$ in matrix form.

Jinfeng
  • 13
0

My formula for the gradient writes

$$2\sum_i (\exp((Ax)_i-b_i))\exp( (Ax)_i)\vec a_i$$ with the column-vector $\vec a_i$ is given by $(\vec a_i)_j = A_{ij}$.

Edit $$\partial_{x_j}\|e^{Ax}-b\|^2 = \sum_i\partial_{x_j}(\exp((Ax)_i)-b_i)^2 $$ $$=2 \sum_i(\exp((Ax)_i)-b_i)\partial_{x_j}\exp((Ax)_i) = 2 \sum_i(\exp((Ax)_i)-b_i) \partial_{x_j}\exp\left(\sum_kA_{ik}x_k\right)$$ $$=2 \sum_i(\exp((Ax)_i)-b_i) \exp\left(\sum_kA_{ik}x_k\right)A_{ij}.$$

TZakrevskiy
  • 22,980