Use the singular value decomposition of $A$ to prove that $||A||_2 = \sigma_1 = \sigma_{\max}(A)$

Question

I have computed the singular value decomposition of the following matrix $$A= \begin{bmatrix}1&2\\0&1\\-1&0\\\end{bmatrix}$$

here are the important findings below.

$$\Sigma=\left[\begin{matrix}1 & 0 \\ 0 & \sqrt 6 \\ 0 & 0\end{matrix}\right]$$

$$V=\left[\begin{matrix}-\frac{2}{\sqrt 5} & \frac{1}{\sqrt 5} \\ \frac{1}{\sqrt 5} & \frac{2}{\sqrt 5}\end{matrix}\right]$$

$$U=\left[\begin{matrix}0 & \sqrt{\frac 5 6} & \frac{1}{\sqrt 6} \\ \frac{1}{\sqrt 5} & \sqrt{\frac 2 {15}} & -\sqrt{\frac 2 3} \\ \frac{2}{\sqrt 5} & -\frac{1}{\sqrt 30} & \frac{1}{\sqrt 6}\end{matrix}\right]$$

Now here is the question I am trying to solve now,

Use the singular value decomposition of $A$ to prove that $\|A\|_2 = \sigma_1 = \sigma_{\max}(A)$. Show further that if $A$ is invertible $\|A^{-1}\|_2 = \sigma_n = \sigma_{\min}(A)$ and thus that the condition number based on the spectral norm is $k(A) = \frac{\sigma_{\max}(A)}{\sigma_{\min}(A)}$

here are my workings for the first part. To find $\|A\|_2=\sigma_1$ use that $\|A\|_2 = \sqrt{\lambda _{\max}(A^TA)}$

$$A^TA =\left[\begin{matrix}2 & 2 \\ 2 & 5 \\\end{matrix}\right]$$ which gives the $$\det(A^TA-\lambda I) = \left[\begin{matrix}2- \lambda & 2 \\ 2 & 5-\lambda \\\end{matrix}\right]$$ resulting in $\lambda = 6,1$ then $\|A\|_2 = \sqrt6 = \sigma_1 = \sigma_{\max}$

I am wondering how I do the next two parts, is the next part the same as I have done above?

Answer 1

Here is an outline of the proof. Various steps are missing justification; you should fill in the blanks. Let $A = U\Sigma V^\top$. $$\|A\|_2^2 = \max_{x : \|x\|_2 = 1} \|Ax\|_2^2 = \max_{x : \|x\|_2 = 1} x^\top A^\top A x = \max_{x : \|x\|_2 = 1} x^\top V \Sigma^\top \Sigma V^\top x = \max_{y : \|y\|_2 = 1} y^\top \Sigma^\top \Sigma y = \max_{y : \|y\|_2 = 1} \sum_{i=1}^r \sigma_i^2(A) y_i^2 = \sigma^2_{\max}(A).$$

If $A$ is invertible, then $\Sigma$ is diagonal and square with nonzero diagonal entries, so $\Sigma^{-1}$ exists and $A^{-1} = V \Sigma^{-1} U^\top$ (why?). Then repeat the above argument to get $\|A^{-1}\|_2 = \sigma_{\min}$.

Answer 2

If $A = U \Sigma V^{T}$ then $U,V$ are orthogonal matrices and $\Sigma $ is a diagonal matrix

The singular values $\sigma_{1} \geq \sigma_{2} \geq \cdots \geq \sigma_{r} > 0 $ are arranged along the diagonal of $\Sigma$ where $A$ has rank $r$ everything else is $0$ then pseudoinverse is given as

$$ A^{\dagger} = V \Sigma^{\dagger} U^{*} \tag{1} $$

$\Sigma^{\dagger} = \begin{align}\begin{cases} \frac{1}{\sigma_{i} } & 1 \leq i \leq r \\ 0 & i > r \end{cases} \end{align} \tag{2} $

Note then if you take the norm of $A^{\dagger}$ you get \begin{align} \| A^{\dagger} \|_{2}^{2} & = \max_{\| x\|_{2} =1 }\| A^{\dagger}x\|_{2}^{2} = \max_{\| x\|_{2} =1 } x^{T} (V \Sigma^{\dagger}U^{T})^{T} (V \Sigma^{\dagger}U^{T}) x \tag{3} \\ & = \max_{\| x\|_{2} =1 } x^{T} U \Sigma^{\dagger}V^{T} V \Sigma^{\dagger}U^{T} x = \max_{\| x\|_{2} =1 } x^{T} U \Sigma^{\dagger}\Sigma^{\dagger}U^{T} x \tag{4}. \end{align}

If you take the transpose of a diagonal matrix you get it back $a_{ii}^{T} = a_{ii}$ $$ \max_{\| y\|_{2} =1 } y^{T} U \Sigma^{\dagger}\Sigma^{\dagger}U^{T} y = \max_{\| y\|_{2} = 1} \sum_{i=1}^{n} \sigma_{i}(A^{\dagger})^{2}y_{i}^{2} = \sigma_{min}^{2}(A) \tag{5}. $$

Note that

$$ \kappa(A) = \| A\|_{2} \| A^{-1}\|_{2} = \frac{\sigma_{max}(A)}{\sigma_{min}(A)} \tag{6}$$

$$ \sigma_{max}(A) = \sqrt{6} \\ \sigma_{min}(A) = 1 \tag{7} $$

$$ \kappa(A) = \frac{\sqrt{6}}{1} \tag{8}$$

if you see the $ \Sigma^{\dagger}$ is $\Sigma $ inverted so when you take the max element you get the minimum element in there.