There are more generic ways of solving such problems.
Since $f(a,b) = ab^2 - ba^2$ is continuous and differentiable, and the domain $0 \le a \le b \le 1$ is a compact simplex, it attains its max and min at one of the following points:
A point where $0 < a < b < 1$ and $\nabla f (a,b) = 0$, i.e. $f_x(a,b) = f_y(a,b) = 0$
A point where $a = 0$, $0 < b < 1$, and $f_y(a,b) = 0$
A point where $0 < a < 1$, $b = 1$, and $f_x(a,b) = 0$
A point where $0 < a = b < 1$, and $f_x(a,b) + f_y(a,b) = 0$
A point where $a = b = 0$ or $a = b = 1$.
Calculate that $f_x(a,b) = b^2 - 2ab$, $f_y(a,b) = 2ab - a^2$.
There are no points of type (1).
Type (2) points are anything on the line, and here $a = 0$ so $f(a,b) = 0$.
Type (3) points satisfy $b = 1$, so $1 - 2a = 0$, so $a = \frac12$, and here $f(a,b) = \frac12 - \frac14 = \frac14$. Type (4) points all have $f(a,b) = 0$.
Finally, the two type (5) points are $f(0,0) = 0$ and $f(1,1) = 0$.
Thus the minimum of $f$ is $0$ and the maximum is $\frac14$, on this domain.