In their working out I understand for the numerator 2+cosx=3 as cosx is less than equal to 1 but in the denominator I don't understand how they got from 3 square root x-x squared sinx all the way upto 2 square root x, I don't understand any of the steps that they have done in the denominator.
Any help would be much appreciated.
The first thing to note is that for $x\in (0,1)$ we have $0<\sin x<1$ so $0<x^2\sin x<x^2$ and therefore $3\sqrt x-x^2\sin x > 3\sqrt x - x^2$.
The next things to note is that for $x\in (0,1)$ we have $\sqrt x >x > x^2 $ and therefore $3\sqrt x - x^2 > 3\sqrt x - \sqrt x = 2\sqrt x$.
So we have $3\sqrt x-x^2\sin x > 2\sqrt x$ as wished.
The question is basically how does the curve behave when x is close to $0$. A rough estimate would be to use cos x = 1 and sin x = x when x is close to $0$.
Then $\frac{2 + \cos x}{3\sqrt x - x^2 \sin x} \approx \frac{3}{3\sqrt x - x^3} \approx \frac{1}{\sqrt x}$ when $x$ is close to $0$ and $\int_0^\epsilon \frac{dx}{\sqrt x}$ is finite.
This is evocative, but it is not proof.
So let's look at $$\frac{1}{\sqrt x} - \frac{2 + \cos x}{3\sqrt x - x^2 \sin x} = \frac{1}{\sqrt x} \left(1 - \frac{2 + \cos x}{3 - x^{3/2} \sin x}\right)$$
Pick some small positive $\epsilon$, say $\epsilon = \frac{1}{10}$ if you insists on a number. For $0 \le x \le \epsilon$,
It follows that
Which gives us $$1 - \frac{2 + \cos x}{3 - x^{3/2} \sin x} < 1 \text{ when } 0 \le x \le \epsilon$$
So $$ 0 < \int_0^{\epsilon} \frac{2 + \cos x}{3\sqrt x - x^2 \sin x}dx < \int_0^{\epsilon} \frac{dx}{\sqrt x} $$
Which is enough to prove that the integral exists.
