Let a,b positive integer
Do you know any closed formulae for the coefficient of $x^k$
in $(x-1)^a(x+1)^b=\sum_{k=0}^{a+b}u(k;a,b)x^k$ ?
I look for an a closed expression of $u(k;a,b)$ involving maybe integral , special function
not the symbol $\sum$
thank's for your help
The coefficient of $x^r$ in $(x-1)^a$ is $\binom ar(-1)^{a-r}$
So, the coefficient of $x^k$ in $(x-1)^a(x+b)^b$ will be $$\sum_{r=0}^k\binom ar(-1)^{a-r}\binom b{k-r}(1)^{b-(k-r)}$$
Starting from lab bhattacharjee's answer, I have been able to remove the $ \Sigma$ and arrived to $$\sum_{r=0}^k\binom ar(-1)^{a-r}\binom b{k-r}(1)^{b-(k-r)}=\frac{(-1)^a \Gamma (b+1) \, _2F_1(-a,-k;b-k+1;-1)}{\Gamma (k+1) \Gamma (b-k+1)}$$
The expansion follows as: \begin{align} (x-1)^{a} (x+1)^{b} &= (-1)^{a} (1-x)^{a} (1+x)^{b} \\ &= (-1)^{a} \sum_{r=0}^{a} \binom{a}{r} (-1)^{r} x^{r} \cdot \sum_{s=0}^{b} \binom{b}{s} x^{s} \\ &= (-1)^{a} \sum_{r=0} \sum_{s=0} (-1)^{r} \binom{a}{r} \binom{b}{s} x^{r+s} \\ &= (-1)^{a} \sum_{r=0}^{a|b} (-1)^{r} \left( \sum_{s=0}^{r} \binom{a}{r-s} \binom{b}{s} (-1)^{s} \right) x^{r}. \end{align} Now \begin{align} \sum_{s=0}^{r} \binom{a}{r-s} \binom{b}{s} (-1)^{s} &= \sum_{s=0}^{r} \frac{a!}{(r-s)!(a-r+s)!} \frac{b!}{s! (b-s)!} (-1)^{s} \\ &= \binom{a}{r} \sum_{s=0}^{r} \frac{1}{(r+1)_{-s} (a+1-r)_{s}} \frac{(-1)^{s}}{s! (b+1)_{-s}} \\ &= \binom{a}{r} \sum_{s=0}^{r} \frac{(-r)_{s} (-b)_{s} (-1)^{s}}{s! (a+1-r)_{s}} \\ &= \binom{a}{r} {}_{2}F_{1}(-r, -b; a-r+1; -1). \end{align} From this the expansion becomes \begin{align} (x-1)^{a} (x+1)^{b} &= \sum_{r=0}^{a|b} \left( (-1)^{r+a} \binom{a}{r} {}_{2}F_{1}(-r, -b; a-r+1; -1) \right) x^{r}. \end{align}
The use of $a|b$ is to mean min(a,b), ie the minimum of $a$ or $b$.
