How to get reduction formula of $$u_n=\int\frac{x^n}{\sqrt{ax^2+2bx+c}}$$
Here $P_{n-1}(x)$ is a polynomial of degree $(n-1)$ $$u_n=\int\frac{x^n}{\sqrt{ax^2+2bx+c}}=P_{n-1}(x)\sqrt{ax^2+2bx+c}+k\int\frac{dx}{\sqrt{ax^2+2bx+x}}$$ Differentiating: $$x^n=P'_{n-1}(x)(ax^2+2bx+c)+\frac12P_{n-1}(x)(2ax+2b)+k$$ $$\implies k=0$$ I don't know how to proceed further:
$$(n+1)u_{n+1}+(2n+1)bu_n+ncu_{n-1}=?$$
You can Proceed in this way:
$$a\, u_{n+2}=\int \frac{ax^{n+2}}{\sqrt{ax^2+2bx+c}}dx$$
$$2b\,u_{n+1}=\int \frac{2bx^{n+1}}{\sqrt{ax^2+2bx+c}}dx$$
$$c\,u_n=\int \frac{cx^{n}}{\sqrt{ax^2+2bx+c}}dx$$
Adding all
$$a\,u_{n+2}+2b\,u_{n+1}+c\,u_n=\int x^n\sqrt{ax^2+2bx+c}dx=v_n \tag{1} $$
Using Integration by Parts to evaluate $$v_n=\int x^n\sqrt{ax^2+2bx+c}dx=(\sqrt{ax^2+2bx+c})\frac{x^{n+1}}{n+1}-\frac{1}{n+1}\int \frac{x^{n+1}(ax+b)} {\sqrt{ax^2+2bx+c}}$$
$\implies$
$$v_n=(\sqrt{ax^2+2bx+c})\frac{x^{n+1}}{n+1}-\frac{a\,u_{n+2}}{n+1}-\frac{b\,u_{n+1}}{n+1}$$
Using above result of $v_n$ in $(1)$ we get
$$a\,u_{n+2}+2b\,u_{n+1}+c\,u_n=(\sqrt{ax^2+2bx+c})\frac{x^{n+1}}{n+1}-\frac{a\,u_{n+2}}{n+1}-\frac{b\,u_{n+1}}{n+1}$$
$\implies$
$$a(n+2)u_{n+2}+b(2n+3)u_{n+1}+c(n+1)u_n=x^{n+1}\sqrt{ax^2+2bx+c}$$
