The validity of the result holds as long as you take your branches correctly. For example, if $\epsilon < a < b$, you can take the branches (blue for $a$ and green for $b$) as

and if $a < \epsilon < b$, then you can take the branches as

In the thick part, the branches cancel, the path is well defined and you can use Cauchy's integral formula without problems,
$$
\oint_{|z| = \epsilon} \frac{\sqrt{(z-a)(z-b)}}{z} dz = 2\pi i \sqrt{(z-a)(z-b)}\Big|_{z = 0} = 2 \pi i \sqrt{ab}
$$
If both $a$ and $b$ are negative, the same argument applies.
Case $a < \epsilon$ and $b < \epsilon$
In this case, you have to take
$\hskip1.3in$
and then see what happens with the small branch going from $a$ to $b$ by taking the contour
$\hskip1.3in$
and make the gap of the external circle go to zero. Then the result will be the contribution of the branch plus the pole.
To find the residue of a function $f(z)$ at $z = a$, we can look at the coefficient of $1/(z-a)$ in the Laurent expansion of $f(z)$ about $z = a$. From WolframAlpha, we can find that the coefficient is $\sqrt{ab}$, and so by the Residue theorem:
$$\oint_{|z|=\epsilon} z^{-1}[(z-a)(z-b)]^{1\over 2}\ dz = 2\pi i \sqrt{ab}.$$
Do branch cuts prevent use of the Residue theorem?
– notes Oct 10 '12 at 19:54