Mathematics Stack Exchange
Mathematics Stack Exchange

Questions tagged [matrix-calculus]

Matrix calculus is about doing calculus, especially derivative and infinite series over spaces of vectors and matrices.

Matrix calculus studies derivatives and differentials of scalar, vector and matrix with respect to vector and matrix. It has been widely applied into different areas such as machine learning, numerical analysis, economics etc.

There are basically two methods.

  • Direct: Regard vectors and matrices as scalar so as to compute in the usual way in calculus. And The Matrix Cookbook provides a lot of basic facts.

  • Component-wise: Write everything in indices notation and compute in the usual way componentwisely. Einstein summation convention is frequently used.

3828 questions
1
vote
1 answer

Gradient of $\ln(\|Ax\|_2^2)$

I want to find gradient of $\ln(\|Ax\|_2^2)$ where $A \in \mathbb{R}^{n \times n} $ and $x \in \mathbb{R}^{n}$. $\ln(\|Ax\|_2^2) = \ln((Ax)^{T}Ax) = \ln(x^TA^TAx)$ And I am stuck here. Trying to do it like in one dimensional case would lead to…
Kate
  • 191
1
vote
1 answer

Integral of product of matrix exponentials

Is there an analytic expression for $$ \int_{0}^{\infty} e^{-At}e^{-Bt} dt $$ where $A$ and $B$ are non-commuting matrices?
Bananach
  • 7,934
1
vote
1 answer

Derivative of a function that takes in a k dimensional vector and outputs a scalar

So we have the function $f(\textbf{B}) = a_0 + \textbf{a}^T\textbf{B}$ and we want to do the following operation$\frac{\partial}{\partial \textbf{B}}$ My intuition tells me this should result in $\textbf{a}^T \textbf{1}$. But according to my…
financial_physician
  • 769
1
vote
2 answers

Is derivative of max(0, x) a diagonal matrix or a vector?

I am reading this amazing tutorial and so far everything was clear and good. Unfortunately, there is this section which doesn't make sense to me: Why is the derivative not a diagonal matrix but a vector? According to this page tanh's derivative is…
Kristof
  • 21
1
vote
1 answer

What is the derivative of matrix multiplication, wrt another matrix?

I'm in a deep learning class, and I always seem to mess up derivative questions, because I put the matrices in the wrong order or transposed/not when they were supposed to be the other way around. Here's one simple question I have, what is: $$\frac{…
Joe
  • 2,661
1
vote
2 answers

How to calculate $\frac{\partial \mathrm{logdet}({X}_{i\times i})}{\partial X}$?

I know that $\frac{\partial \mathrm{logdet}(X)}{\partial X}=(X^{-1})^T$. Lets denote ${X}_{i\times i}$ which is $i\times i$ matrix that is formed from $X\in\mathbb{R}^{n\times n}$ by removing all columns $i+1,i+2,\ldots,n$ and rows…
Lee
  • 1,910
  • 12
  • 19
1
vote
2 answers

Please explain $\frac{\partial}{\partial a_{ij}} \sum_{i=1}^m a_{1i} b_{i1} + \cdots + \sum_{i=1}^m a_{ni} b_{in} = b_{ji}$

I need a detailed step by step to understand please. This is one step from the broader proof 1 of $\nabla_A \mathrm{tr} AB=B^T$ whose preceding steps to this point I understand. This is a totally new area for me so please be explicit in detail. The…
Joe
  • 489
1
vote
1 answer

Chain Rule for Matrix Valued Derivatives with intermediate high dimensional tensors

I have one function $f: R^{m \times m} \to R$ and another function $g: R \to R^{m \times m}$. I would like to calculate the derivate $\frac{\partial h}{\partial x}$ for the composite function $h(x) = f(g(x))$. The expression I find is really messy…
Tom Bennett
  • 540
1
vote
1 answer

matrix raised to a matrix

i wanted to know if it was possible to raise a matrix to a matrix and i wanted to confirm if i have it. i tested it out with numbers and notation and i want to know if i'm right. $$ x =\left [ \begin{matrix} 1 & 1 \\ 0 & 1 \\ …
Pie Pie
  • 29
1
vote
0 answers

Gradient of $f = \gamma \ \left( x - 1\mu(x) \right) \ \left( \sigma(x) + \epsilon \right)^{-1/2} + 1\beta$ w.r.t. $x$, $\gamma$, and $\beta$

How to compute the gradient of $$\eqalign{ f &= \gamma \ \left( x - 1\mu(x) \right) \ \left( \sigma(x) + \epsilon \right)^{-1/2} + 1\beta \cr \mu(x) &= \alpha \ 1^Tx \cr \sigma(x) &= \alpha \sum_{k=1}^m \left( x_k - \mu(x) \right)^2 \equiv \alpha…
learning
  • 661
1
vote
1 answer

Hadamard product tricks of particular entities

Consider the following matrices $\mathbf{Q}_{H}$ order $\left( T\times n\right) $ and $\mathbf{A}$ of order $\left( T\times T\right) $ and $% \mathbf{\hat{u}}$ of order $\left( n\times 1\right) $ and denote by $\circ $ the Hadamard product and by…
Anonymous
  • 310
1
vote
1 answer

Derivative of $max(0, \mathbf{x})$ for the vector $\mathbf{x} \in R^n$

I'm reading the tutorial The Matrix Calculus You Need For Deep Learning: https://arxiv.org/abs/1802.01528. In Page 25, the derivative of the ReLu function $\text{max}(0, \mathbf{x})$, where the variable $\mathbf{x}$ is a vector $\in R^n$, is given…
user159566
  • 97
  • 6
1
vote
1 answer

Gradient of matrix valued function

I am wondering if its possible to obtain an analytic expression for the gradient of $$f(B) = (A - B)\left[(A - B)'(A - B)\right]^{-\frac{1}{2}}$$ with respect to $B$ where $A \in \mathbb{R}^{s \times t}$, $B \in \mathbb{R}^{s \times t},$ and $(A -…
user23658
  • 453
1
vote
2 answers

matrix: solve for square matrix with 0 diagonal

I have an equation of $XI = Y$, where I know $I$ (vector of 1s) and $Y$ (vector of positive integers). I want to find positive square matrix $X$, I also know that the diagonal entries of $X$ are all 0. So example is $X$ is 3x3, $I$ is 3x1 and $Y$…
Dirk N
  • 123
1
vote
1 answer

need help with matrix calculus

I am trying to find $\frac{\partial (x'Ax)}{\partial x}$ where x is a vector (2 x 1 vector) and A is a matrix (say 2x2 dimensions). When I looked up in http://www.matrixcalculus.org/ I found the answer to be $(A.x)' + x'.A$ where $'$ stands for…
Upendra01
  • 133
1 2 3
…
13 14