Some statistical (learning) issues

Question

im reading about statistical learning (Trevor Hastie, Robert Tibshirani, Jerome Friedman The elements of statistical learning) and for some reason it seems to be trivial that $E[XX^T]$ is non-singular (with $X \in \mathbb{R}^P$ a random real valued vector) but I dont really think its that easy, am i missing something here?

Answer 1

This will only be a problem if $X$ has a really pathological distribution. Firstly, let's remap $X$ to have zero mean, i.e. $X \leftarrow X - \mathbb{E}[X]$; then $C=\mathbb{E}[XX^T]$ is exactly the covariance matrix of $X$, which should tell you that it is positive semi-definite at least. We can show this by taking $v\in\mathbb{R}^P$, then $$ v^TCv= v^T\mathbb{E}[XX^T ]v = \mathbb{E}[v^T X X^T v] = \mathbb{E}[v^T X (v^T X)^T] = \mathbb{E}[\alpha^T\alpha] = \mathbb{E}[\alpha^2] \geq 0 $$ where $\alpha = v^T X= v\cdot X$ and $v\ne \vec{0}$. So it's positive semi-definite.

But when is $\alpha = 0$ (the problem case)?

Well, $v$ is non-zero, but $X$ might be, but how often does that happen? Assuming $X$ is not a Dirac Delta distribution at zero, then in practice probably never, especially if $P$ is high. This is because the probability that a continuous random variable takes a specific value is zero (also here), under mild assumptions.

Of course, the real issue here is that we need to ensure $X$ is never orthogonal to $v$. This happens when all the probability mass (density) for $X$ lies on a hyperplane that is orthgonal to $v$, i.e. the data lies on $$ X^Tv=0 $$ In other words, our distribution cannot lie on a low dimensional subspace of the data space. This is exactly the problem people sometimes find with degenerate features in their dataset: it results in sample covariance matrices with zero determinant (i.e., your matrix would be singular). Check out these posts: [1], [2], [3].

Again, however, for any reasonable distribution, this cannot happen. The reason is that hyperplanes have measure zero (see also here), meaning choosing a random vector will essentially never hit the plane (for the same reason as the probability of a continuous random variable hitting an exact value is zero).

In any case, you would have to choose $X$ to very carefully adhere to a hyper-plane in dataspace, which doesn't occur in practice unless you do something like exactly duplicate your features. So ruling that out, yes, our $C$ is safe (non-singular) and positive definite with probability $1$, in theory.

One caveat is when working with sample covariance matrices $\widehat{C}$ on a computer. A combination of small sample size, and floating point and round-off error (e.g., dealing with data of small magnitude) can lead to the occasional singularity, if the distribution $p(X)$ is close to being degenerate. (Often when trying to compute $\log \det C$ for instance). One can fix this by adding a tiny bit of noise to the data, or by adding a tiny perturbation $\epsilon I_P$ to $\widehat{C}$.