84

I have a very simple question that can be stated without any proof. Are all eigenvectors, of any matrix, always orthogonal? I am trying to understand principal components and it is crucial for me to see the basis of eigenvectors.

Fermion
  • 17
Bober02
  • 2,546
  • 11
    No. Take any any non-orthogonal basis $(v_1,\dots,v_n)$ and define a linear map $A$ on this basis by sending each $v_i$ to $iv_i$. The eigenspaces are the $n$ lines generated by the $v_i$, and these are by construction not ortgogonal. – Olivier Bégassat May 08 '12 at 13:38
  • 16
    Eigenvectors corresponding to different eigenvalues will be orthogonal if the matrix is symmetric. This is part of the real spectral theorem. – Dylan Moreland May 08 '12 at 13:38
  • 2
    The case @Dylan is describing will apply to your study of principal components, since the underlying matrices are symmetric... – J. M. ain't a mathematician May 08 '12 at 14:50
  • @PL this is clearly not a duplicate of that question - this question is about the general case when $M$ is not necessarily a real symmetric matrix. – KReiser Jul 26 '20 at 01:55

5 Answers5

76

In general, for any matrix, the eigenvectors are NOT always orthogonal. But for a special type of matrix, symmetric matrix, the eigenvalues are always real and eigenvectors corresponding to distinct eigenvalues are always orthogonal. If the eigenvalues are not distinct, an orthogonal basis for this eigenspace can be chosen using Gram-Schmidt.

For any matrix $M$ with $n$ rows and $m$ columns, $M$ multiplies with its transpose, either $M M'$ or $M'M$, results in a symmetric matrix, so for this symmetric matrix, the eigenvectors are always orthogonal.

In the application of PCA, a dataset of $n$ samples with $m$ features is usually represented in a $n\times m$ matrix $D$. The variance and covariance among those $m$ features can be represented by a $m\times m$ matrix $D'D$, which is symmetric (numbers on the diagonal represent the variance of each single feature, and the number on row $i$ column $j$ represents the covariance between feature $i$ and $j$). The PCA is applied on this symmetric matrix, so the eigenvectors are guaranteed to be orthogonal.

giobrach
  • 7,490
Yue Tyler Jin
  • 861
  • 1
    The qualification "symmetric" for a matrix should almost always be accompanied by "real", in cases where the notion is useful; that is the case for this answer. Also, one does not need the Gram-Schmidt procedure to choose an orthogonal basis; it is only useful to correct the inadvertent choice of a non-orthogonal basis. If in the end you need an orthogonal basis, just add the condition of orthogonality to your system every time you choose a basis vector before choosing another one, and you basis will be orthogonal straight away. – Marc van Leeuwen Jul 30 '23 at 09:34
33

Fix two linearly independent vectors $u$ and $v$ in $\mathbb{R}^2$, define $Tu=u$ and $Tv=2v$. Then extend linearly $T$ to a map from $\mathbb{R}^n$ to itself. The eigenvectors of $T$ are $u$ and $v$ (or any multiple). Of course, $u$ need not be perpendicular to $v$.

Siminore
  • 35,136
  • Thank you very much for your answers. Could anyone state whether they are orthogonal in PCA case? – Bober02 May 08 '12 at 13:45
  • 2
    For PCA, things can always be set up such that the eigenvectors are orthogonal. On the other hand, I would recommend looking at PCA as a singular value decomposition instead of as an eigendecomposition. It's been discussed here on math.SE a number of times; search around. – J. M. ain't a mathematician May 08 '12 at 14:48
  • What do you mean by "setting up"? Is there some common technique to achive a singular matrix? – Bober02 May 09 '12 at 08:12
  • 3
    My understanding based on https://en.wikipedia.org/wiki/Principal_component_analysis is that a singular value decomposition (SVD) of $X$ results in three matrices, where the first of them is composed of the eigenvectors of $X^T X$, and is used for PCA. This matrix is always orthogonal. – Uri Jun 14 '13 at 13:04
  • Beautiful answer. – Don Larynx Nov 13 '13 at 15:14
  • what is PCA? Hmm... So that's a counter example. – user4951 Jun 06 '16 at 14:15
9

Not necessarily all orthogonal. However two eigenvectors corresponding to different eigenvalues are orthogonal, whenever the matrix is symmetric.

e.g Let $X_1$ and $X_2$ be two eigenvectors of a matrix $A$ corresponding to eigenvalues $\lambda_1 $ and $\lambda_2$ where $\lambda_1 \ne \lambda_2$.

Now,

$AX_1 = \lambda_1 X_1$ and $AX_2 = \lambda_2 X_2$.

Taking Transpose of first,

$ \begin{align} &(AX_1)^T = (\lambda_1 X_1)^T\\ \implies & X_1^TA^T = \lambda_1 X_1^T\\ \implies &X_1^TA^TX_2 = \lambda_1 X_1^T X_2\\ \implies & X_1^T\lambda_2 X_2 = \lambda_1 X_1^T X_2\\ \implies &\lambda_2X_1^T X_2 = \lambda_1 X_1^T X_2\\ \implies &(\lambda_2 - \lambda_1)X_1^T X_2 = 0\\ \end{align} $

Since $\lambda_2 \ne \lambda_1$, $X_1^T X_2 $ must be $0$. This establishes orthogonality of eigenvectors corresponding to two different eigenvalues.

Marine Galantin
  • 2,956
  • 1
  • 16
  • 33
Jai Vrat Singh
  • 123
4

In the context of PCA: it is usually applied to a positive semi-definite matrix, such as a matrix cross product, $X ' X$, or a covariance or correlation matrix.

In this PSD case, all eigenvalues, $\lambda_i \ge 0$ and if $\lambda_i \ne \lambda_j$, then the corresponding eivenvectors are orthogonal. If $\lambda_i = \lambda_j$ then any two orthogonal vectors serve as eigenvectors for that subspace.

user101089
  • 499
0

Not only are the eigenvectors not generally orthogonal, they're not always uniquely defined. For example, any basis whatever for the space constitutes a set of eigenvectors for the eigenvalue 1 of the identity matrix.

You can orthogonalize the eigenvectors to taste --- but orthogonality plays no role in their being eigenvectors.

As the other posters pointed out, there are special classes of matrices where orthogonality is involved in the eigenanalysis.

Steve White
  • 31
  • 4