$A,B$ are complex matrices in $M(n,\mathbb{C})$ and rank$(AB)$=rank$(BA)$=rank$(ABA)$. Prove that $AB$ is similar to $BA$.

Question

$A,B$ are complex matrices in $M(n,\mathbb{C})$ and rank$(AB)$=rank$(BA)$=rank$(ABA)$. Prove that $AB$ is similar to $BA$.

I tried by using JCF..but could not complete

Answer 1

The main idea is to use your rank information to get a detailed view of the Jordan structure of eigenvalue zero. The Jordan structure for the non-zero eigenvalues take care of themselves (see end).

The Frobenius Rank Inequality says
$\text{rank}\big(XY \big) +\text{rank}\big( YZ\big) \leq \text{rank}\big(XYZ\big) + \text{rank}\big(Y\big)$

we also know
(i) $\text{rank}\big(A(BA)^{m}\big)=\text{rank}\big((BA)^{m}\big)$
for all natural numbers $m$
because $\text{span}\Big((BA)^{m}\Big)\subseteq \text{span}\Big((BA)\Big)$
and $A$ is injective on the vector (sub)space generated by $\text{span}\big((BA)\big)$
Note: this also implies for all natural numbers $m$
$\text{rank}\Big((AB)^{m+1} \Big)=\text{rank}\Big(B(AB)^m\Big)=\text{rank}\Big((BA)^mB\Big)$
(why?)

Base Case
$Z:= A$
$X:=B$
$Y:=AB$
and apply Frobenius Rank Inequality
$\text{rank}\big(BAB \big) +\text{rank}\big(AB \big)=\text{rank}\big(B(AB) \big) +\text{rank}\big((AB)A\big) \leq \text{rank}\big(B(AB)A\big) + \text{rank}\big(AB \big)$
$\implies \text{rank}\big(BAB \big)\leq \text{rank}\big(BABA\big)\leq \text{rank}\big(BAB \big)$
(by submultiplicativity of rank on the right)

we conclude
$\text{rank}\big((AB)^2\big)= \text{rank}\big(BAB \big)=\text{rank}\big((BA)^2\big)$
(where the LHS follows by (i), setting $m:=1$)

Inductive Case
(ii) the induction hypothesis is
$\text{rank}\big((AB)^{k}\big)=\text{rank}\Big(B(AB)^{k-1}\Big)= \text{rank}\big((BA)^k\big)$
for $k\geq 2$
and need to show this implies
$\text{rank}\big((AB)^{k+1}\big)=\text{rank}\Big(B(AB)^{k}\Big)= \text{rank}\big((BA)^{k+1}\big)$

To prove this, apply Frobenius Rank Inequality
$X:=B$, $Y:= (AB)^k$, $Z:=A$
$\text{rank}\Big(B(AB)^k \Big) +\text{rank}\Big((AB)^kA\Big) \leq \text{rank}\Big(B(AB)^k A\Big) + \text{rank}\Big((AB)^k \Big)$

an equivalent statement is
$\text{rank}\Big(B(AB)^k \Big) +\text{rank}\Big(A(BA)^k\Big) \leq \text{rank}\Big((BA)^{k+1}\Big) + \text{rank}\Big((AB)^k \Big)$
$\implies\text{rank}\Big(B(AB)^k \Big) +\text{rank}\Big((BA)^k\Big) \leq \text{rank}\Big((BA)^{k+1}\Big) + \text{rank}\Big((AB)^k \Big)$ (by (i))
$\implies \text{rank}\Big(B(AB)^k \Big)\leq \text{rank}\Big((BA)^{k+1}\Big)\leq \text{rank}\Big((BA)^kB\Big) = \text{rank}\Big(B(AB)^k \Big)$ (by (ii))
$\implies \text{rank}\Big((BA)^{k+1}\Big) = \text{rank}\Big(B(AB)^{k}\Big)=\text{rank}\Big((AB)^{k+1}\Big) $
where the RHS follows by (i) and completes the induction

Since $\text{rank}\Big((AB)^{m}\Big) = \text{rank}\Big((BA)^{m}\Big) $
for all natual numbers $m$, we know that $(AB)$ and $(BA)$ have the same Jordan structure with respect to eigenvalue $0$. And by AB and BA have identical nonsingular Jordan blocks they have the same Jordan structure with respect to non-zero eigenvalues.
Thus $(AB)$ and $(BA)$ are similar to the same jordan matrix $J\implies (AB)$ and $(BA)$ are similar.

---

alternative to induction
one way to bypass the Inductive Case is to just note that the Base Case tells us
$\text{rank}\Big(BAB \Big)=\text{rank}\Big((BA)^2\Big)=\text{rank}\Big((BAB)A\Big)$
i.e. it tells us that $A^T$ is injective on the space generated by $\text{span}\Big((BAB)^T\Big)$

but then for all natural numbers $k$
$\text{rank}\Big(B(AB)^{k}\Big)= \text{rank}\Big((BA)^k B\Big)= \text{rank}\Big((BA)^{k-1}(BAB)\Big)= \text{rank}\Big((BAB)^T\big((BA)^{k-1}\big)^{T}\Big)= \text{rank}\Big(A^T(BAB)^T\big((BA)^{k-1}\big)^{T}\Big)= \text{rank}\Big(\big((BA)^{k-1}\big)(BAB)A\Big) = \text{rank}\Big((BA)^{k+1}\Big)$
since transposition doesn't change rank and because $\text{span}\Big((BAB)^T\big((BA)^{k-1}\big)^{T}\Big)\subseteq \text{span}\Big((BAB)^T\Big)$
and $A^T$ is injective on the space generated by RHS, hence injective on any subspace. This is a slightly faster route but still rather ugly.

Answer 2

Let $r=\operatorname{rank}(A)$. Then $X:=PAQ=\pmatrix{I_r&0\\ 0&0}$ for some invertible matrices $P$ and $Q$. Let $Y=Q^{-1}BP^{-1}$. Then $$ XY=PABP^{-1},\ YX=Q^{-1}BAQ\ \text{ and }\ XYX=PABAQ. $$ It follows that $\operatorname{rank}(XY)=\operatorname{rank}(YX)=\operatorname{rank}(XYX)$ and it suffices to show that $XY$ is similar to $YX$. Let $$ Y=\pmatrix{E&F\\ G&H} $$ where $E$ is $r\times r$. The condition $\operatorname{rank}(XY)=\operatorname{rank}(YX)=\operatorname{rank}(XYX)$ thus implies that $$ \operatorname{rank}\pmatrix{E&F}=\operatorname{rank}\pmatrix{E\\ G}=\operatorname{rank}(E). $$ Therefore $F=EZ$ and $G=WE$ for some matrices $Z$ and $W$. Hence \begin{aligned} XY&=\pmatrix{E&EZ\\ 0&0}=\pmatrix{I&-Z\\ 0&I}\pmatrix{E&0\\ 0&0}\pmatrix{I&Z\\ 0&I}\text{ and}\\ YX&=\pmatrix{E&0\\ WE&0}=\pmatrix{I&0\\ W&I}\pmatrix{E&0\\ 0&0}\pmatrix{I&0\\ -W&I} \end{aligned} are similar to each other and we are done.