The problem is to determine the integer values $n\geq 2$ such that there exist matrices $A,B\in M_n(\mathbb{C})$ with the properties $(AB)^2=(BA)^2=O_n$ and $ABA\ne O_n, BAB\ne O_n$.
Apparently, for $n=2$ no such matrices exist. Also, if one finds such a value $n$, then the matrices $A,B$ can be completed with zeros to generate matrices from $M_k(\mathbb{C})$ with the requested property for any $k\geq n$.
The given conditions are satisfied by $$ A=\pmatrix{1&0&0&0\\ 0&1&0&0\\ 0&0&1&0\\ 0&0&0&0} \ \text{ and }\ B=\pmatrix{0&0&1&0\\ 0&0&0&1\\ 0&0&0&0\\ 0&1&0&0}. $$ By padding the above $A$ and $B$ with zeroes, the conditions are also satisfied for all $n\ge 4$.
No feasible $A$ and $B$ exist when $n\le3$. Suppose the contrary. View $A$ as the matrix representation of a linear map from an $n$-dimensional vector space $V$ to another $n$-dimensional vector space $W$. View $B$ as the matrix representation of a linear map from $W$ to $V$. By changing the bases in $V$ and $W$ separately, we may assume that $$ A=\pmatrix{I_r&0\\ 0&0} \ \text{and}\ B=\pmatrix{X&Y\\ Z&W} $$ where $X$ is $r\times r$. Perform some further changes to the bases, we may also assume that $X$ in Jordan form. The given conditions can then be rewritten as a set of five conditions: \begin{aligned} X^2&=0,\\ ZX&=0,\\ XY&=0,\\ X&\ne0,\\ ZY&\ne0. \end{aligned} The last two conditions thus imply that $X,Y,Z$ are non-empty. Hence $0<r<n$.
If $r=1$, then $X$ is a scalar and the conditions $X^2=0$ and $X\ne0$ contradict each other.
Hence $r=2$ and $n=3$. Since $X^2=0$ and $X\ne0$ and $X$ is in Jordan form, we have $X=\pmatrix{0&1\\ 0&0}$. The conditions $ZX=0$ and $XY=0$ then imply that $Z=\pmatrix{0&\ast}$ and $Y=\pmatrix{\ast\\ 0}$. But then the condition $ZY\ne0$ is violated. Thus we conclude that no feasible $A$ and $B$ exist when $n\le3$.
