I'm trying to prove that the set of orthogonal matrices $O(n)$ is a manifold. (We identify the set of matrices of order $n$ with $\Bbb R^{n^2}$) In order to do so, I must show $f'$ has maximal rank in $O(n)$, having already defined $f:\Bbb R^{n^2}\longrightarrow\Bbb R^{n^2}$ by $f(A)=AA^t-I_n$.
How can I do that? I proved it with $n=2,3$, but I don't know how to generalize the argument. A hint is enough, since this is part of a problem set.
Do you know about the equivariant rank theorem?
One way I know to prove this (from Lee's Smooth Manifolds) is to define the map to be
$F(A) = A^TA$
which is a map from the general linear group $GL(n)$ to the symmetric group $S(n)$. The orthogonal group is then $F^{-1}(I)$.
Then, the equivariant rank theorem says
Let $M$ and $N$ be smooth manifolds and let $G$ be a Lie group. Suppose $F: M \to N$ is a smooth map that is equivariant with respect to a transitive smooth $G$-action on $M$ and any smooth $G$-action on $N$. Then $F$ has constant rank.
Equivariant means $F(g\cdot p) = g\cdot F(p)$ (for a left action) or $F(p \cdot g) = F(p) \cdot g$ (for a right action).
You can take $M = GL(n)$, $N = S(n)$, and $G = GL(n)$. At this point, you may want to try coming up with the actions on your own, but I'll put them below in a spoiler tag in case you get stuck.
Let $GL(n)$ act on $M$ (i.e. itself) by right multiplication (it's clear why this is transitive, right?), and on $N$ by conjugation, i.e. $X \cdot B = B^TXB$. Then equivariance follows: $F(A \cdot B) = F(AB) = (AB)^T(AB) = B^TA^TAB = B^TF(A)B = F(A) \cdot B$
Baron VT's answer is more appropriate for your manifold focus, but here is a direct proof anyway:
It is is easy to check directly that $Df(A)(\Delta) = L(\Delta) = A \Delta^T +\Delta A^T$.
Using the rank nullity theorem we can focus on $\ker L$ rather than $\operatorname{rk} L$
Note that $A$ is invertible, and so the map $\phi(\Delta) = \Delta A$ is invertible (in fact, $\phi^{-1}(\Delta) = \Delta A^T$ ).
Let $L'(\Delta) = L(\Delta A) = \Delta^T+\Delta$. We note that $\ker L'$ is exactly the set of skew-symmetric matrices, which has dimension $\frac{1}{2}n(n-1)$, and hence $\dim \ker L = \frac{1}{2}n(n-1)$ also.
