I know it is possible to have such a matrix if $A$ is $3\times 3$. I think it won't be possible in this case but I'm not sure how to prove a general case.
A specific case proof would work something like this.
$A$ is determined by its action on $e_1,e_2$
Say $Ae_1=\alpha e_1 + \beta e_2$.
Note that $\alpha=0$ else $A^3\ne0$
Similarly $Ae_2=\gamma e_1$
Hence both $\beta=0=\gamma$
How to prove for larger dimension? That is, $A\in M_{n\times n}$ then it is not possible that $A^n\ne0=A^{n+1}$