By definition, we are required to show that, for each $\epsilon>0$, there is some $\delta>0$ such that, for all points (x,y), if $|(x,y)-(0,0)|<\delta$, then $|5x^3-x^2y^2-0|<\epsilon$. In particular, we must be careful to avoid any dependencies between x and y, so as not to inadvertently miss important limit subsets in more pathological cases.
The $\delta$ inequality is equivalent to $\sqrt{x^2+y^2}<\delta$, so we may conveniently use polar coordinates to deduce our requirements, by defining $r=\sqrt{x^2+y^2}$, as well as $x=r\cos\theta$ and $y=r\sin\theta$.
As in most $\epsilon-\delta$ proofs, we start at the inequality we want to be true, then work backwards to find the necessary restrictions on $\delta$. Then we present the forwards implications using the found $\delta$.
First, let us rewrite the inequality in polar coordinates. We see that we require $|5r^3\cos^3(\theta)-r^4\cos^2(\theta)\sin^2(\theta)|<\epsilon$.
By the triangle inequality, we know that $|5r^3\cos^3(\theta)-r^4\cos^2(\theta)\sin^2(\theta)| \leq 5r^3|\cos^3(\theta)|+r^4\cos^2(\theta)\sin^2(\theta)$.
Since $\cos^2(\theta)\sin^2(\theta)\leq 1$, we also have $5r^3|\cos^3(\theta)|+r^4\cos^2(\theta)\sin^2(\theta)\leq 5r^3|\cos^3(\theta)|+r^4$.
Likewise, since $|\cos^3(\theta)|\leq 1$, we have $5r^3|\cos^3(\theta)|+r^4\leq 5r^3+r^4$.
If $r\geq 1$, then $5r^3+r^4\leq 5r^4+r^4=6r^4$. If $6r^4<\epsilon$, then $\frac{\epsilon}{6}>1$ and $r<\left(\frac{\epsilon}{6}\right)^\frac{1}{4}$.
If, on the other hand, $\frac{\epsilon}{6}<1$, then $r<1$ and $r^4+5r^3<r^3+5r^3$, and we have $6r^3<\epsilon$. It suffices to choose $r<\frac{\epsilon}{6}$ in this case.
Now we may begin the formal proof.
For each $\epsilon > 0$, let $\delta \leq \min\left(\frac{\epsilon}{6},\left(\frac{\epsilon}{6}\right)^\frac{1}{4}\right)$.
Then, starting with $|5r^3\cos^3(\theta)-r^4\cos^2(\theta)\sin^2(\theta)|$ and working through the inequalities as above, we come to the expression $5r^3+r^4$.
If $\epsilon\geq 6$, then $\frac{\epsilon}{6}\geq\left(\frac{\epsilon}{6}\right)^\frac{1}{4}$ and therefore $r<\left(\frac{\epsilon}{6}\right)^\frac{1}{4}$. Thus, $5r^3+r^4 < 5\left(\frac{\epsilon}{6}\right)^\frac{3}{4} + \frac{\epsilon}{6}$.
Since $\frac{\epsilon}{6}\geq 1$, we have $\left(\frac{\epsilon}{6}\right)^\frac{3}{4}\leq \frac{\epsilon}{6}$, so $5\left(\frac{\epsilon}{6}\right)^\frac{3}{4} + \frac{\epsilon}{6}\leq 5\frac{\epsilon}{6} + \frac{\epsilon}{6} = \epsilon$.
Likewise, if $\epsilon < 6$, then $r<\frac{\epsilon}{6}<1$ implies that $5r^3+r^4 < 5r^3 + r^3 = 6r^3 = \epsilon$.
As always, if you are overly concerned about using rectangular coordinates, we may simply replace $r$, $\cos\theta$ and $\sin\theta$ with the appropriate expressions.