2

could anyone explain:

a) $||{Ax-b}||^2$ (there is also a lowered 2): what does this two 2's mean?

b) why is the solution: $x =(A^TA)^{-1} A^Tb$ is?

Thank you very much:)

Ethan Bolker
  • 95,224
  • 7
  • 108
  • 199
lars111
  • 321
  • The least square method minimizes the square of the $2$-norm of $Ax - b$. The solution is the projection of $b$ on the subspace spanned by the columns of $A$. – Hubble Nov 25 '15 at 17:52
  • The lowered 2 you're mentioning declares the type of norm you're considering. For example $|x|_2$ is the euclidean norm, more or less the most standard case. It is defined for $x\in\mathbb{R}^n$ as

    $|x|_2 = \sqrt{x_1^2 + x_2^2 + ... + x_n^2}$.

     – Denis Düsseldorf Nov 25 '15 at 17:53

1 Answers1

1

Start with the linear system $$ \mathbf{A}x = b, \quad \mathbf{A}\in\mathbb{C}^{m\times n}, \quad x \in \mathbf{C}^{n}, \quad b \in \mathbf{C}^{m}. $$ Assume $\mathbf{A}$ has full column rank where the number of columns is equal to the rank $\rho$, that is $n=\rho$. If there is no $x\in\mathcal{R}(\mathbf{A})$ that solves this equation, then instead of trying to make the residual error vector $r(x) = \mathbf{A}x - b$ be zero, make it as small as possible.

To measure size and distance, we must choose a norm. You have chosen the $2-$norm as elucidated by @Denis Düsseldorf. This is the familiar norm of Pythagoras: $$ \lVert x + y \rVert_{2}^{2} = x^{2} + y^{2} $$

There are a variety of ways to solve the least squares problem. Your question is about the normal equations: $$ \mathbf{A}^{*} \mathbf{A} x = \mathbf{A}^{*} b. $$ By construction, this system is consistent. The vector $\mathbf{A}^{*} b$ is clearly in the range space of $\mathbf{A}^{*}$. Because we have a full column rank linear system, the $n\times n$ product matrix $\mathbf{A}^{*} \mathbf{A}$ has full rank, and is nonsingular. Therefore the inverse exists and the least squares solution is $$ x_{LS} = \left( \mathbf{A}^{*} \mathbf{A} \right)^{-1} \mathbf{A}^{*} b. $$

dantopa
  • 10,342