The original question is:
Show that the matrix $\Omega^{-1/2}X(X'\Omega^{-1}X)^{-1}X'\Omega^{-1/2}$, where $X$ is an $n\times p$ matrix and $\Omega^{-1}=\Omega^{-1/2}\Omega^{-1/2}$, is idempotent. What is its rank?
The first part is easy to show. But what's its rank?
Thanks in advance!
First note that, in order for $X^T \Omega^{-1} X$ to have the inverse, $X$ has to be of a full column rank, i.e., $X$ has rank $p \le n$.
Let us get some idea of the solution. We'll assume $\Omega$ to be symmetric (for the root to exist, this also means it's positive definite). Let $\Omega^{-1/2}X = U \Sigma V^T$ be an SVD of $\Omega^{-1/2}X$ Then
$$\Omega^{-1/2} X (X^T \Omega^{-1} X)^{-1} X^T \Omega^{-1/2} = U \Sigma V^T (V \Sigma^T \Sigma V^T)^{-1} V \Sigma^T U^T = U \Sigma (\Sigma^T \Sigma)^{-1} \Sigma^T U^T$$
also has rank $p$. This is most likely our solution., so let's get it, using a similar approach.
Let us assume that $\Omega$ is just nonsingular (not necessarily symmetric) such that $\Omega^{-1/2}$ exists. Let $X = U \Sigma V^T$ be an SVD of $X$.
We'll need some additional notation. Let $\tilde{\Sigma}$ denote the top left block of $\Sigma$ of order $p$, i.e.,
$$\Sigma = \begin{bmatrix} \tilde{\Sigma} \\ 0 \end{bmatrix}.$$
Furthermore, let
$$I_\Sigma = \begin{bmatrix} I_{p \times p} \\ 0_{(n-p) \times p} \end{bmatrix}.$$
Then $\Sigma = I_\Sigma \tilde{\Sigma}$ and $\Sigma^T = \tilde{\Sigma} I_\Sigma^T$. Note that $\tilde{\Sigma}$ is symmetric and nonsingular. Now,
\begin{align*} M &:= \Omega^{-1/2} X (X^T \Omega^{-1} X)^{-1} X^T \Omega^{-1/2} \\ &= \Omega^{-1/2} U \Sigma V^T (V \Sigma^T U^T \Omega^{-1} U \Sigma V^T)^{-1} V \Sigma^T U^T \Omega^{-1/2} \\ &= \Omega^{-1/2} U \Sigma ( \Sigma^T U^T \Omega^{-1} U \Sigma )^{-1} \Sigma^T U^T \Omega^{-1/2} \\ &= \Omega^{-1/2} U I_\Sigma \tilde{\Sigma} ( \tilde{\Sigma} I_\Sigma^T U^T \Omega^{-1} U I_\Sigma \tilde{\Sigma} )^{-1} \tilde{\Sigma} I_\Sigma^T U^T \Omega^{-1/2} \\ &= \Omega^{-1/2} U I_\Sigma ( I_\Sigma^T U^T \Omega^{-1} U I_\Sigma )^{-1} I_\Sigma^T U^T \Omega^{-1/2}. \end{align*}
So, $M$ has the same rank as $I_\Sigma ( I_\Sigma^T U^T \Omega^{-1} U I_\Sigma )^{-1} I_\Sigma^T$ which is a block matrix of a form
$$\begin{bmatrix} (I_\Sigma^T U^T \Omega^{-1} U I_\Sigma)^{-1} & 0 \\ 0 & 0 \end{bmatrix}.$$
Since $(I_\Sigma^T U^T \Omega^{-1} U I_\Sigma)^{-1}$ is nonsingular of order $p$, we conclude that the rank of $M$ is $p$.
Since $X'\Omega^{-1}X$ is an invertible $p\times p$ matrix, each factor in the product must have rank at least $p$. In particular, $n\ge p$. Since $X'$ is $p\times n$ and has rank $p$, its range is all of $K^p$, where $K$ is the underlying field (maybe $\mathbb{R}$).
It should be clear that the range of $X'\Omega^{-1/2}$ is $K^p$. Now following that with an invertible map $K^p\to K^p$, we conclude that the range of $(X'\Omega^{-1}X)^{-1}X'\Omega^{-1/2}$ is $K^p$ as well. So the range of $X(X'\Omega^{-1}X)^{-1}X'\Omega^{-1/2}$ is a $p$-dimensional subspace of $K^n$, since $X$ must be injective. Finally, $\Omega^{-1/2}$ is invertible, so it maps this $p$-dimensional space to another $p$-dimensional space, and it follows that the range of $\Omega^{-1/2}X(X'\Omega^{-1}X)^{-1}X'\Omega^{-1/2}$ is $p$-dimensional.
