Let $a = re^{i \theta}$
Then, $a = b^2 = (a^2)^2 = a^4 = r^4e^{i (4 \theta)}$
What we need is, first $r=r^4$
That is, $r =0$ or $r = 1$ (since $r$ must be real)
$r=0$ is trivial, discard that.
Next, we need $e^{i \theta} = e^{i(4 \theta)}$
That is, $4\theta = \theta$ or $4 \theta = \theta + 2n\pi, n\in\mathbb Z\setminus \{0\}.$
You were missing this $\boldsymbol{2 n \pi}$ part in your attempt.
From here, we get $\theta =0 $ or $3\theta = 2n\pi$
$\theta=0$ means real numbers, discard that.
That leaves us with $3\theta = 2n\pi$
For $n=1, \theta = 2\pi/3$
For $n=2, \theta = 4\pi/3$
Other $n$ values can be discarded as for those $\theta \not \in [0, 2 \pi)$
Thus we're left with $e^{i (2\pi/3)}$ and $e^{i (4\pi/3)}$
They are both solutions, and these are the only non-real solutions for $a = b^2$ and $b = a^2$. They are the two complex cube roots of unity.
In real numbers, $a=b=0$ and $a=b=1$ form another solutions as we discovered above. The entire solution set in complex numbers is
$(a, b) \in \{(0,0), (1,1),
(\omega, \omega^2),
(\omega^2, \omega)\}$
where $\omega = e^{i (2\pi/3)}$