Exactly what is the definition of pdf of negative binomial distribution? So I have two textbooks one said: $$f_X(x)=\begin{pmatrix} r+x-1\\ x \end{pmatrix}p^rq^x$$ where as on the other it is defined as $$f_X(x)=\begin{pmatrix} r+x-1\\ x \end{pmatrix}p^{r-1}q^x$$ So which one should we use?
Also, What is that x-1 for? cause you have r times success and x times fail, so isn't the total number of trial just r+x?
Thanks in advance.
The total number of trials is indeed $r+x$.
There are several closely related but not identical definitions of the negative binomial.
Definition 1: The random variable $T$ is the number of trials until and including the $r$-th success.
Definition 2: The random variable $U$ is the number of trials up to but not including the $r$-th success.
Definition 3: The random variable $V$ is the number of failures up to the $r$-th success.
I prefer Definition 3. Elementary texts usually use Definition 1.
Let us find $\Pr(T=t)$ under Definition 1. The $r$-th success occurs on the $t$-th trial if (i) There are exactly $r-1$ successes in the first $t-1$ trials and (ii) there is a success on the $t$-th trial.
The probability of (i) is $\binom{t-1}{r-1}p^{r-1}(1-p)^{t-r}$, and given that (i) happened the probability of (ii) is $p$, giving $$\binom{t-1}{r-1}p^r (1-p)^{t-r}.$$ Note that this is defined for $t\ge r$ only.
Definition 2 gives a very similar formula. The probability that $U=u$ is equal to the probability that $T$ is $u+1$.
For Definition 3, the same sort of reasoning gives that the number $V$ of failures before the $r$-th success is given by $$\Pr(V=v)=\binom{r+v-1}{r-1}p^r (1-p)^v.$$
Remark: Both formulas you quote seem to be interested in the number of failures. Note that $\binom{r+x-1}{x}=\binom{r+x-1}{r-1}$, so that the first expression you quote gives the same result as the one discussed under Definition 3.
The second quoted formula is not correct. However, as discussed under Definition 3, it is the main step towards a solution.
