The negative binomial distribution is a discrete distribution so has a probability mass function (as you have stated - note there are other parametrisations of the negative binomial) rather than a density. It is the Beta distribution which has a density, proportional to $p^a(1-p)^b$.
If you have a single observation $x$ from your negative binomial distribution with $x \in \{r,r+1,\ldots\}$ then the posterior density for $p$ given $x$ will be (ignoring multiplicative constants) proportional to $p^a (1-p)^b p^r (1-p)^{x-r}$ when $0 <p<1$, with the first part coming from the Beta prior and the second part from the negative binomial likelihood.
Tidying this up gives a posterior density proportional to $p^{a+r}(1-p)^{b+x-r}$, which is the density of a Beta distribution with parameters $a+r$ and $b+x-r$.
If instead you had $n$ observations, the posterior distribution would be a Beta distribution with parameters $a+nr$ and $b+(\sum x_j)-nr$.