You can solve it via hyperbolic substitution plus integration by parts (it'd be too cumbersome to work only with IBP).
We begin by doing $x=a\sinh\theta$, thus $dx=a\cosh\theta d\theta$.
$$
\begin{aligned}
\mathcal J\equiv\int x^2\sqrt{a^2+x^2}dx&=\int a^2\sinh^2\theta\sqrt{a^2+a^2\sinh^2\theta}\ a\cosh\theta d\theta\\
&=a^4\int\sinh^2\theta\cosh^2\theta d\theta\\
&=\dfrac{a^4}{4}\int\sinh^22\theta d\theta.
\end{aligned}$$
Now we proceed with IBP in order to compute $\int\sinh^22\theta d\theta$.
We choose $u=\sinh2\theta$ and $dv=\sinh2\theta d\theta$, meaning that $du=2\cosh\theta d\theta$ and that $v=\frac{1}{2}\cosh2\theta \ (+C)$.
$$
\begin{aligned}
\frac{4}{a^4}\mathcal J=\mathcal I=\int\sinh^22\theta d\theta&=\frac{1}{2}\sinh 2\theta\cosh2\theta-\int\cosh^22\theta d\theta\\
&=\frac{1}{2}\sinh 2\theta\cosh2\theta-\int(1+\sinh^22\theta)d\theta\\
&=\frac{1}{2}\sinh 2\theta\cosh2\theta-\theta-\mathcal I
\end{aligned}$$
$$\implies \mathcal I=\frac{1}{2}\sinh 2\theta\cosh2\theta-\theta-\mathcal I$$
$$\hspace{0.5cm}\implies\mathcal I=\frac{1}{4}\sinh 2\theta\cosh2\theta-\frac{\theta}{2}.$$
At last we revert the substitution expressing $\theta$ in function of $x$ $\left(\theta=\sinh^{-1}\left(\frac{x}{a}\right)\right)$. Thus, the final result should be
$$\mathcal J=\frac{a^4}{16}\sinh\left[2\sinh^{-1}\left(\frac{x}{a}\right)\right]\cosh\left[2\sinh^{-1}\left(\frac{x}{a}\right)\right]-\frac{a^4}{8}\sinh^{-1}\left(\frac{x}{a}\right)+C.$$
(Maybe it can be further simplified in the form of radicals?)
