A is Non Square matrix, where '$m$' rows < '$n$' columns.
For example $m=10$; $n=15$;
When calculating $(\textbf A \cdot\textbf A ^ T) ^ {-1}$ results a singular matrix and inverse fails.
So how to overcome this? So that, I can multiply $\textbf A^T$ with above equation?
I am not so familiar with matrices. I heard that we can do this with eigen values/ vectors. Please do the needful
I Got Inverse by Singular Value Decomposition & Principal Component Analysis concept by Jonathan Harel http://www.klab.caltech.edu/~harel/
Suppose we have an m x n matrix A, in which we interpret the COLUMNS to be sample vectors. For any of the m components of each sample vector, we can define the sample variance (we assume zero-mean data), and we wish to find direction in $R{^m}$ which have maximal variance over the set. The idea is that a few such directions span a subspace of $R{^m}$ which captures most of the variance in the data. We can then project a new vector onto this subspace as means of reducing dimensionality.
If the rows of A sum to zero, then the covarience matrix is an m x m matrix C such that:
C = E[*${a_i}$${a_j}$]* = const. $AA{^T}$
where $a{_i}$ and $a{_j}$ are the $i{^{th}}$ and $j{^{th}}$ components (i,j $w\in \{1,2,.., m\}$) from the same randomly selected sample vector, and const = 1/n .We will drop this leading constant from here on as it does not play a role in the analysis. The variance of the data in A projected onto an arbitary direction v turns out to be :
var(v) = $v^T$Cv = $\sum_{i,j}$$v_i$$v_j$$C_{ij}$. (1)
Since variance must be positive, then $v^TCv>0$ for all real nonzero v, and because C is also symmetric, C must be positive semi-definite.
According to optimization theory, the variance quantity (1) is maximized(subject to $v^Tv=1$) When
grad($v^TCv$) = $\lambda$.grad($v^Tv$)[grad wrt components of v]
=> Cv = $\lambda v$
Therefore, the direction with maximum variance is an eigenvector of C. We note that the variance of an eigenvector is
$e^TCe=e^T\lambda e = \lambda$
So the line along the principal eigenvector of C is the one-dimentional subspace in which the data (projected onto such a subspace) in A have the gretest variance. The direction orthogonal to that eigenvector with largest variance among all possible orthogonal directions is given by the second eigenvector, etc.. The eigenvectors are orthogonal because C is symmetric.
Thus, if
C=$U \wedge U^T$ is the eigendecomposition of C, then
A=$U \sqrt{\wedge V^T}$ is the singular value decomposition of A.
Coded in R(http://www.ats.ucla.edu/stat/r/pages/svd_demos.htm):
library(MASS)
A <- matrix(c(1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1), 9, 4)## 9 rows n 4 columns
A.svd <- svd(A)
A.svd$\$d$ ##The non-zero singular values of A
Result of A.svd$\$d$ :-
## [1] 3.46e+00 1.73e+00 1.73e+00 1.92e-16
ds <- diag(1/A.svd$\$d$[1:3]) ## variance along the principal component
u <- A.u ##left-singular vectors of a A
v <- A.v ##left-singular vectors of a A
us <- as.matrix(u[, 1:3]) ## Convert to matrix format
vs <- as.matrix(v[, 1:3]) ## Convert to matrix format
(A.ginv <- vs %*% ds %*% t(us)) ## Inverse Step
## [,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8]
## [1,] 0.0833 0.0833 0.0833 0.0833 0.0833 0.0833 0.0833 0.0833
## [2,] 0.2500 0.2500 0.2500 -0.0833 -0.0833 -0.0833 -0.0833 -0.0833
## [3,] -0.0833 -0.0833 -0.0833 0.2500 0.2500 0.2500 -0.0833 -0.0833
## [4,] -0.0833 -0.0833 -0.0833 -0.0833 -0.0833 -0.0833 0.2500 0.2500
## [,9]
## [1,] 0.0833
## [2,] -0.0833
## [3,] -0.0833
## [4,] 0.2500