We start by considering two sequences -
$$ A \rightarrow x_{n+1} = \frac{x_{n}}{2}(1 + \frac{a}{x_{n}^2})$$
and
$$ B \rightarrow x_{n+1} = \frac{x_{n}}{2}(3 - \frac{x_{n}^2}{a}) $$
We can observe that by letting $\lim_{n \rightarrow \infty} x_{n} = \xi$ and $\lim_{n \rightarrow \infty} x_{n+1} = \xi$ , $\xi^2 = a$ where $\xi$ is the exact solution.
Next we substitute $x_{n} = \xi + \epsilon_{n}$ and $x_{n+1} = \xi + \epsilon_{n+1}$ and $a = \xi^2$ to both $A$ and $B$ and we obtain the following equations respectively.
In case of
$$A : \epsilon_{n+1} = \frac{\epsilon_{n}^2}{2 \xi} + O(\epsilon^3)$$
which implies that iti s 2nd oreder convergent with error constant $\frac{1}{2\xi}$.
for the case
$$B : \epsilon_{n+1} = \frac{-3}{2\xi} (\epsilon^{n})^2 + O(\epsilon_{n}^2)$$
We next do this operation - adding three times equation $A$ to equation $B$,in order to get third order convergence as after this operation $\epsilon_{n+1} = O(\epsilon_{n}^3)$.
so after the above operation we get the following equation
$$x_{n+1} = \frac{1}{8}x_{n}(6 + 3\frac{a}{x_{n}^2} - \frac{x_{n}^2}{a})$$
and comparing to your given formula
we get the coefficients $\alpha = \frac{1}{8} , \beta = \frac{3}{8}.$