When in equilibrium the plane containing the rim of the hemisphere makes an angle with the vertical, call it $\psi$, say.
The centre of the sphere, the point of contact $P$, and the centre of the hemisphere $O$ all lie in the same vertical line.
Let the particle, situated at point $A$, the lowest point of the rim, have mass $m$ and the hemisphere mass $M$. The centroid $G$ of the hemisphere is at a distance $\frac12a$ from $O$ on the axis of symmetry of the hemisphere.
Let the vertical line $OP$ intersect $AG$ at $B$, so that $h=BP$ the distance you seek. Let $\angle OAB=\alpha$, so $\tan\alpha=\frac12$.
Taking moments at $P$ we have $$mga\sin\psi=Mg \frac a2 \cos\psi\implies \tan\psi=\frac{M}{2m}$$
Then by the Sine Rule in $\triangle OAB$, we have $$\frac{a-h}{\sin\alpha}=\frac{a}{\sin(\alpha+\psi)}$$
Using the compound angle identity and rearranging, you get $$h=\frac{a\cos\psi+2a\sin\psi-a}{\cos\psi+2\sin\psi}$$
If you now make the assumption (which you have not justified) that if $\frac1h>\frac1a+\frac1b$, there is stable equilibrium, substituting for $h$ into this inequality leads to $$\frac ba>\cos\psi+2\sin\psi-1$$
Since we can write $\cos\psi+2\sin\psi$ in the form $\sqrt{5}\sin(\psi+\epsilon)$ which has a maximum value of $\sqrt{5}$ for all $\psi$, which is determined by the masses, then we can deduce the result required.
On the other hand, if you were to do this problem without making the assumption mentioned you would have to do the following:
Allow the hemisphere to roll without slipping through a small angle $\phi$ from the position of equilibrium. The point of contact $P'$ is now such that, if $C$ is the centre of the sphere, $OP'C$ is a straight line and the angle which $OP'C$ now makes with the vertical is $\theta$, and these two angles are related by the equation $$b\theta=a\phi$$
The vertical height of $O$ above $C$ is now $$(a+b)\cos\theta$$
The vertical depth of $A$ below $O$ is now $$a\cos(\theta+\phi+\psi)$$
The vertical depth of $G$ below $O$ is now $$\frac a2\sin(\theta+\phi+\psi)$$
With this information it is straightforward to write down an expression, after eliminating $\phi$, for the potential energy of the system $V$, as a function of $\theta$, taking fixed point $C$ as the zero level.
If you differentiate this once, it is easy to verify that at $\theta=0$, we get $\frac{dV}{d\theta}=0$ thus indicating equilibrium at $\theta=0$ as expected.
To establish the required inequality, you would then set $$\frac{d^2V}{d\theta^2}>0$$ since this is the requirement for stable equilibrium, and sure enough, after a bit of algebra, you arrive at $$\frac ba>\cos\psi+2\sin\psi-1,$$ and hence the result.
You might like to try this yourself.