First of all, Both answers $I=-\frac{\cos ^3 x}{3}+\frac{\cos ^5 x}{5}+C= \frac{1}{80} \cos 5 x-\frac{1}{48} \cos 3 x-\frac{1}{8} \cos x+C$ are correct, which can be confirmed by the following proof:
Let $z=\cos x+i\sin x$, then $\sin x= \frac{1}{2i}(z-\frac{1}{z})$ and $\cos x= \frac{1}{2}(z+\frac{1}{z})$.
Expressing $\sin^3x\cos^2x$ in terms of $z$ and then sine of multiples of $x$ gives
$$
\begin{aligned}
\sin ^3 x \cos ^2 x =&-\frac{1}{32 i}\left(z-\frac{1}{z}\right)^3\left(z+\frac{1}{z}\right)^2 \\
=&-\frac{1}{32 i}\left(z^2-\frac{1}{z^2}\right)^2\left(z-\frac{1}{z}\right) \\
=&-\frac{1}{32 i}\left[\left(z^5-\frac{1}{z^5}\right)-\left(z^3-\frac{1}{z^3}\right)-2\left(z-\frac{1}{z}\right)\right] \\
=&-\frac{1}{32 i}(2 i \sin 5 x-2 i \sin 3 x-4 i \sin x) \\
=&-\frac{1}{16} \sin 5 x+\frac{1}{16} \sin 3 x+\frac{1}{8}\sin x
\end{aligned}
$$
Integrating back yields
$$
\boxed{\int \sin ^3 x \cos ^2 x d x=\frac{1}{80} \cos 5 x-\frac{1}{48} \cos 3 x-\frac{1}{8} \cos x+C}
$$
Wish it helps!