It seems fairly straightforward. The substitution is: $$x=x'\cos\theta-y'\sin\theta,y=x'\sin\theta+y'\cos\theta$$ That gives the coefficient of the $x'y'$ term on the lhs as $$-4\cos\theta\sin\theta+\sqrt3(\cos^2\theta-\sin^2\theta)+2\cos\theta\sin\theta
$$ The idea is to choose the constant $\theta$ to make this term vanish. So we need $$2\cos\theta\sin\theta=\sqrt3(\cos^2\theta-\sin^2\theta)$$ You then use the double angle formulae to get $$\sin2\theta=\sqrt3\cos2\theta$$ Incidentally, this gives a rotation of the coordinate axes, rather than a translation.
------- added later
More detail on the substitution. We start with $2x^2+\sqrt3 xy+y^2=8$ and we make the substitution above. The first term becomes $$2x^2=2(x'\cos\theta-y'\sin\theta)^2$$ $$=2x'^2\cos^2\theta-4x'y'\cos\theta\sin\theta+2y'^2\sin^2\theta\quad(1)$$ The second term becomes $$\sqrt3 xy=\sqrt3(x'\cos\theta-y'\sin\theta)(x'\sin\theta+y'\cos\theta)$$ $$=\sqrt3 x'^2\sin\theta\cos\theta+x'y'\sqrt3(\cos^2\theta-\sin^2\theta)-y'^2\sqrt3\sin\theta\cos\theta\quad(2)$$ The third term becomes $$y^2=(x'\sin\theta+y'\cos\theta)^2$$ $$=x'^2\sin^2\theta+2x'y'\cos\theta\sin\theta+y'^2\cos^2\theta\quad(3)$$ We are interested in the $x'y'$ terms. Adding these three terms together we get the result above.
------- added later still
We start with the equation $$2x^2+\sqrt3\ xy+y^2=0$$ We make the substitutions. The three terms turn out as above at (1), (2), (3). So the equation turns into the sum of the right-hand sides of (1), (2), (3) equals 0. If we add these right-hand sides we can rearrange to get a term in $x'^2$, a term in $x'y'$ and a term in $y'^2$. We are only interested in the term in $x'y'$ because we want that term to be 0. So from each of (1), (2), (3) we pick out the $x'y'$ term and we add them. (1) gives $$-4x'y'\cos\theta\sin\theta$$ (2) gives $$x'y'\sqrt3(\cos^2\theta-\sin^2\theta)$$ and (3) gives $$2x'y'\cos\theta\sin\theta$$
Adding these gives $$-2x'y'\cos\theta\sin\theta+x'y'\sqrt3(\cos^2\theta-\sin^2\theta)$$ [Note that $-4x'y'\cos\theta\sin\theta$ and $2x'y'\cos\theta\sin\theta$ add to give $-2x'y'\cos\theta\sin\theta$.] That takes us to the fourth line of this answer.
Note that I have not worked out the final result of adding (1), (2), (3), because we don't care about the coefficients of $x'^2$ and $y'^2$. All we care about is finding the right value of $\theta$ to eliminate the $x'y'$ term.