$\cos^2\theta=\dfrac{m^2-1}{3}$, $\tan^3\dfrac{\theta}{2}=\tan\alpha$. How to prove $$\cos^\frac{2}{3}\alpha+\sin^\frac{2}{3}\alpha=\frac{2}{m}^{2/3}. $$ I got the following
$$\tan^\frac{2}{3}\alpha=\frac{1- \sqrt\frac{m^2 - 1}{3}}{1+\sqrt\frac{m^2 - 1}{3}} $$ but i can not proceed further.
This is one of those problems where a step that seems like it could never work somehow does. Start with your equation and add one to both sides.
$$ \tan^{\frac23}\alpha +1 = \frac{1- \sqrt{\frac{m^2-1}{3}}}{1+ \sqrt{\frac{m^2-1}{3}}}+1$$
Multiply both sides by $\cos^{\frac23}\alpha$ to get $$ \sin^{\frac23}\alpha + \cos^{\frac23}\alpha= \frac{2*\cos^{\frac23}\alpha}{1+\sqrt{\frac{m^2-1}{3}}}.$$
This is closer, but that cosine term is still there. Unfortunate, but lets blindly chug along and see what happens. Going back to the equation you gave, we can solve for $\cos^2\alpha$. Cube both sides and substitute for $\sin^2\alpha$ to get $$ \frac{1-\cos^2\alpha}{\cos^2\alpha} = \left(\frac{1- \sqrt{\frac{m^2-1}{3}}}{1+ \sqrt{\frac{m^2-1}{3}}} \right)^3.$$ Now split the left side into $\frac{1}{cos^2\alpha} - 1$ and add one to both sides of the equation. Next, invert and manipulate the expression slightly to get the following expression for $\cos^2\alpha$: $$ cos^2\alpha = \frac{\left(1+ \sqrt{\frac{m^2-1}{3}}\right)^3}{\left(1+ \sqrt{\frac{m^2-1}{3}}\right)^3 + \left(1- \sqrt{\frac{m^2-1}{3}}\right)^3}$$
Now the special thing that makes this problem work is the denominator of this expression. The two terms are identical except for the minus sign. This means that the 2nd and 4th terms of the binomial expansion cancel out and we are left with $$ \left(1+ \sqrt{\frac{m^2-1}{3}}\right)^3 + \left(1- \sqrt{\frac{m^2-1}{3}}\right)^3 = 1+1 + 3\left(\frac{m^2-1}3\right)+3\left(\frac{m^2-1}3\right) = 2m^2.$$ This is the kind of thing that only happens in homework problems. With the denominator simplified, we can take the cube root of $cos^2\alpha$ and get $$ \cos^{\frac23}\alpha = \frac{1+ \sqrt{\frac{m^2-1}{3}}}{2^{\frac13}m^{\frac23}}.$$ Plus this back into the expression for $cos^{\frac23}\alpha + sin^{\frac23}\alpha$ and simplify to get the desired relation.
$$\tan\alpha=\tan^3\dfrac\theta2$$
$$\iff\frac{\sin\alpha}{\sin^3\dfrac\theta2}=\frac{\cos\alpha}{\cos^3\dfrac\theta2}=\pm\sqrt{\frac{\sin^2\alpha+\cos^2\alpha}{\sin^6\dfrac\theta2+\cos^6\dfrac\theta2}}$$
$$\iff\frac{\sin\alpha}{\sin^3\dfrac\theta2}=\frac{\cos\alpha}{\cos^3\dfrac\theta2}=\pm\frac1{\sqrt{\left(\sin^2\dfrac\theta2\right)^3+\left(\cos^2\dfrac\theta2\right)^3}}\ \ \ \ (1)$$
Now $\left(\sin^2\dfrac\theta2\right)^3+\left(\cos^2\dfrac\theta2\right)^3=\left(\sin^2\dfrac\theta2+\cos^2\dfrac\theta2\right)^3-3\left(\sin^2\dfrac\theta2+\cos^2\dfrac\theta2\right)\left(\sin^2\dfrac\theta2\cos^2\dfrac\theta2\right)$
$=1-\dfrac34\sin^2\theta=\cdots $
Take $2/3$ power of each sides of $(1)$
$$\dfrac{\tan^2\theta/2-1}{\tan^2\theta/2+1}=\cos^2\theta=\dfrac{m^2-1}{3}\iff\tan^2\theta/2=-\left(\dfrac{m^2+2}{m^2-4}\right)=\tan^{\frac{2}{3}}\alpha\to(*)$$ We know that $$\cos^2\alpha+\sin^2\alpha=(\cos^\frac{2}{3}\alpha+\sin^\frac{2}{3}\alpha)(\cos^\frac{4}{3}\alpha-\sin^\frac{2}{3}\alpha\cos^\frac{2}{3}\alpha+\sin^\frac{4}{3}\alpha)=1.$$ Threfore it is enough to calculate $$\cos^\frac{4}{3}\alpha-\sin^\frac{2}{3}\alpha\cos^\frac{2}{3}\alpha+\sin^\frac{4}{3}\alpha=\cos^\frac{4}{3}\alpha(1-\tan^{\frac{2}{3}}\alpha+\tan^{\frac{4}{3}}\alpha)=\left(\dfrac{\tan^2\alpha}{1+\tan^2\alpha}\right)^{\frac{2}{3}}(1-\tan^{\frac{2}{3}}\alpha+\tan^{\frac{4}{3}}\alpha)\to(**)$$ Substitute $(*)$ into $(**)$ and simplify, then you will get your banswer.
