I have the taylor expansion:
$$f(x+h)=f(x)+hf'(x)+ \frac 1 2 h^{2}f''(x)+O(h^3) $$
and I am trying to get the cubic iteration of the the Taylor method:
$ x_{n+1}=x_{n}-f'(x_{n})(1-\frac{\sqrt{(1-2f"(x_{n})f(x_{n})})}{f"(x_{n})} )$
I have tried setting $f(x+h)=0$ and solving for h but it doesn't seem to be working
Improving on Donald Splutterwit, you want the smaller of the two solutions, so \begin{align} h&=-\frac{f'(x)}{f''(x)}\left(1-\sqrt{1-2\frac{f(x)f''(x)}{f'(x)^2}}\right)\\ &=-\frac{f'(x)}{f''(x)}\frac{2\frac{f(x)f''(x)}{f'(x)^2}}{1+\sqrt{1-2\frac{f(x)f''(x)}{f'(x)^2}}}\\ &=-\frac{2f(x)}{f'(x)}\left(1+\sqrt{1-2\frac{f(x)f''(x)}{f'(x)^2}}\right)^{-1} \end{align} which is Halley's original method. Only later was the square root replaced by the linear Taylor polynomial to get the "modern" Halley method that can also be written as $$ x_+=x+\frac{d(1/f)^{(d-1)}(x)}{(1/f)^{(d)}(x)} $$ for $d=2$ ($d=1$ gives the usual Newton method).
Another method to get a third order $h$ is to multiply $$ 0=f(x)+(f'(x)+\tfrac12f''(x)h)h+O(h^3) $$ with $(f'(x)-\tfrac12f''(x)h)$ to get $$ 0=f(x)f'(x)+\Bigl(f'(x)^2-\tfrac12f(x)f''(x)\Bigr)h+O(h^3) $$ which now is linear in $h$ and can be solved to $$ h=-\frac{f(x)f'(x)}{f'(x)^2-\tfrac12f(x)f''(x)} $$ which is again the increment of the Halley method.
Let $f(x+h)=0$, so we have $h^2f''(x)+2hf'(x)+2f(x)=0$. Now solve this as a quadratic in $h$ \begin{eqnarray*} h= \frac{-f'(x) \pm \sqrt{ (f'(x))^2 -2 f(x)f''(x)}}{f''(x)} \end{eqnarray*} Now let $h=x_{n+1}-x_n$ and $x=x_n$ and we have the desired result.
