Trigonometric substitution seems easier here. We may set $x = \tan\frac{A}{2}, y = \tan\frac{B}{2}, z = \tan\frac{C}{2}$, where $A, B, C$ are angles of a triangle, so that the constraint is satisfied. Then we have to find the maximum of
$$f = \dfrac{1}{1+\tan^2\frac{A}{2}}+\dfrac{1}{1+\tan^2\frac{B}{2}}+\dfrac{\tan\frac{C}{2}}{1+\tan^2\frac{C}{2}} = \cos^2\dfrac{A}{2} + \cos^2\dfrac{B}{2}+ \dfrac{\sin C}{2}$$
Using $\cos \theta = 2 \cos^2 \frac{\theta}{2} - 1$,
$$f = \dfrac{\cos A + 1}{2} + \dfrac{\cos B + 1}{2}+ \dfrac{\sin C }{2} = 1+\dfrac{\cos A + \cos B + \sin C}{2}$$
So we need to find the maximimum of $\cos A + \cos B + \sin C$, where $A, B, C$ are angles of a triangle. Now,
$$\cos A + \cos B + \sin C = \cos A + \cos B + \sin (\pi - (A+B)) \\
= \frac{2}{\sqrt3}\left(\frac{\sqrt3}{2} \cos A + \frac{\sqrt3}{2} \cos B\right) + \frac{1}{\sqrt3}\left(\sqrt3 \sin A \cos B + \sqrt 3 \cos A \sin B\right)\\
\le \frac{1}{\sqrt3}\left(\frac{3}{4} + \cos^2 A + \frac{3}{4}+ \cos^2 B\right) + \frac{1}{2\sqrt3}\left(3 \sin^2 A +\cos^2 B + \cos^2 A +3\sin^2 B\right)\\
= \frac{\sqrt3}{2}+\frac{\sqrt3}{2}(\sin^2 A + \cos^2A) + \frac{\sqrt3}{2}(\sin^2 B + \cos^2 B) = \frac{3\sqrt 3}{2}$$
Hence $f \le 1 + \frac{3\sqrt3}{4}$, and equality is obtained when the AM-GMs we used have equality, i.e. $A = B = \frac{\pi}{6}, C = \frac{2\pi}{3}$ or $x = y = 2 - \sqrt{3}, z = \sqrt{3}$