I think you have made a mistake in your computations as I did the arithmetic with my CAS. However, if we make the substitutions
$$\begin{align}
a &= 2 \cos t + \frac{2}{\sqrt{3}} \sin t = \frac{4}{\sqrt{3}} \sin(t+\frac{\pi}{3}) \\
b &= 2 \cos t - \frac{2}{\sqrt{3}} \sin t = \frac{4}{\sqrt{3}} \sin(t+\frac{2\pi}{3})
\end{align}$$
then the constraint equation
$$a^2+b^2-ab=4$$
will be satisfied identically $4=4$. Also, to satisfy $a \gt 0$ and $b \gt 0$ we require that
$$\begin{cases}
0 \lt t+\frac{\pi}{3} \lt \pi \\
0 \lt t+\frac{2\pi}{3} \lt \pi
\end{cases}
\to
\begin{cases}
-\frac{\pi}{3} \lt t \lt \frac{2\pi}{3} \\
-\frac{2\pi}{3} \lt t \lt \frac{\pi}{3}
\end{cases}
\to
-\frac{\pi}{3} \lt t \lt \frac{\pi}{3}$$
So, it remains to work on $f(a,b)=2a+b- 3ab$. Now, if we do the substitution in $f(a,b)$ we will get
$$\begin{align}g(t) &= f(a(t),b(t)) \\
&=\frac{2}{\sqrt{3}}\sin t + 6 \cos t - 16 \cos^2 t +4 \\
\end{align}$$
Finally, we can use any method in calculus to find the Maximum and Minimum of this continuous function on the interval $(-\frac{\pi}{3},\frac{\pi}{3})$. So the range of $f(a,b)$ with the desired constraints on $a$ and $b$ will be obtained. In fact, we just reduced a constrained multi-variable optimization problem to a usual single variable optimization problem in calculus.