about a difficult and weird Probability question

Question

Let W be the random variable that counts the number of tails before one gets r heads for a coin whose probability of heads is θ. Without using moment generating function, show that the mean and variance for W are [r(1-θ)]/θ and [r(1-θ)]/θ^2 Please help me with the complete proof. I have some ideas, but I can't write it in a formal way. Thanks!

Answer 1

Let $X$ be the random number of tails that occur before $r$ heads are observed. The probability that $X = x$ is given by the expression $$\Pr[X = x] = \binom{x + r - 1}{x} \theta^r (1-\theta)^x, \quad x = 0, 1, 2, \ldots,$$ because:

1. The total number of coin tosses must be $x + r$;
2. The final coin toss must be heads and must be the $r^{\rm th}$ occurrence of a head;
3. There are $\binom{x+r-1}{x}$ ways to arrange the $x$ tails among the $x+r-1$ tosses before the final toss;
4. The $r$ heads have a joint probability of $\theta^r$ of occurring and the $x$ tails have a joint probability of $(1-\theta)^x$ of occurring.

If we trust that $$\sum_{x=0}^\infty \Pr[X = x] = 1$$ for the PMF above, then it is a straightforward exercise to compute $$\sum_{x=0}^\infty x\Pr[X = x]$$ and $$\sum_{x=0}^\infty x(x-1) \Pr[X = x],$$ the first of which immediately gives the expected value, and the second indirectly leads to the variance with the help of the expectation.

---

As for evaluating the sum, note $$\begin{align*} \sum_{x=0}^\infty x \binom{x+r-1}{x} \theta^r (1-\theta)^x &= \sum_{x=1}^\infty \frac{(x+r-1)!}{(x-1)! \, (r-1)!} \theta^r (1-\theta)^x \\ &= \sum_{x=1}^\infty r \cdot \frac{(x+r-1)!}{(x-1)! \, r!} \theta^r (1-\theta)^x \\ &= r \sum_{x=0}^\infty \frac{(x+r)!}{x! \, r!} \theta^r (1-\theta)^{x+1} \\ &= r \sum_{x=0}^\infty \binom{x+(r+1)-1}{x} \theta^r (1-\theta)^{x+1} \\ &= r \frac{1-\theta}{\theta} \sum_{x=0}^\infty \binom{x+(r+1)-1}{x} \theta^{r+1} (1-\theta)^x, \end{align*}$$ and since this last sum is equal to $1$ (being the sum of the probabilities of all outcomes for a negative binomial distribution with parameter $r+1$ instead of $r$), the result follows. A similar approach applies for the second sum, which I leave to you as an exercise.

Answer 2

Let's apply the binomial theorem to see that $$ (1-x)^{-n-1} = \sum_{k\geq 0} (-1^k) \binom{-n-1}{k} x^k $$ We can negate the upper index in the binomial coefficient, incorporating the $(-1)^k$, to get $$ \frac{1}{(1-x)^{(n+1)}} = \sum_{k\geq 0} \binom{n+k}{k} x^k = \sum_{k\geq 0} \binom{n+k}{n} x^k $$ and if we multiply that by $z^n$ and note that binomial coefficients with integer top less than the integer bottom are zero, we get our key identity

$$ \frac{x^n}{(1-x)^{(n+1)}} = \sum_{k\geq 0} \binom{k}{n} x^k $$

Consider $k$ to be the total number of tosses until the $r$-th head turns up on the $k$-th toss. (Obviously $k \geq r$.) The probability of that happening is $$f(k;r) = \binom{k-1}{r-1} \theta^{r-1} (1-\theta)^{k-r} \theta $$ which we can read off as " $r-1$ out of the first $k-1$ tosses came up heads; the remaining $k-r$ were tails; and then the last toss was the $r$-th head."

$$f(k;r) = \left( \frac{\theta}{1-\theta} \right)^r \binom{k-1}{r-1} (1-\theta)^k $$

Now let's calculate the expectation of the total number of tosses (not just the tails): $$E(k;r) = \sum_{k\geq r} k \left( \frac{\theta}{1-\theta} \right)^r \binom{k-1}{r-1} (1-\theta)^k = \left( \frac{\theta}{1-\theta} \right)^r \sum_{k\geq r} k \binom{k-1}{r-1} (1-\theta)^k $$

But it is easy to see that $k \binom{k-1}{r-1} = r \binom{k}{r}$ by incorporating the $k$ factor into the numerator and then seeing how to ket to a binomial coefficient by multiplying the denominator (by $r$). $$E(k;r) = = \left( \frac{\theta}{1-\theta} \right)^r r \sum_{k\geq r} \binom{k}{r} (1-\theta)^k $$ upon which we apply our key identity (with $x = (1-\theta)$)to get $$E(k;r) = \left( \frac{\theta}{1-\theta} \right)^r r \frac{(1-\theta)^r}{\theta^{r+1}} = \frac{r}{\theta} $$ The mean number of tails is then $$\frac{r}{\theta} - r = r \frac{1-\theta}{\theta}$$ as was to be shown.

The variance of the number of tails is the same as the variance of the total number of tosses, as these differ by a fixed constant $r$. This in turn is $E(k^2;r) - \left( E(k;r) \right)^2$.

$$ E(k^2;r) = \left( \frac{\theta}{1-\theta} \right)^r \sum_{k\geq r} k^2 \binom{k-1}{r-1} (1-\theta)^k = \left( \frac{\theta}{1-\theta} \right)^r \left[ \sum_{k\geq r} (k+1)k \binom{k-1}{r-1} (1-\theta)^k - \sum_{k\geq r} k \binom{k-1}{r-1} (1-\theta)^k \right] $$ $$= \left( \frac{\theta}{1-\theta} \right)^r \left[ r(r+1) \sum_{k\geq r} \binom{k+1}{r+1}(1-\theta)^k - r \sum_{k\geq r}\binom{k}{r} (1-\theta)^k \right] = \left( \frac{\theta}{1-\theta} \right)^r \left[ r(r+1) \sum_{m > r} \binom{m}{r+1}(1-\theta)^{m-1} - r \frac{(1-\theta)^r}{\theta^{r+1}} \right]=\left( \frac{\theta}{1-\theta} \right)^r \left[ r(r+1) \frac{1}{1-\theta} \sum_{m > r} \binom{m}{r+1}(1-\theta)^m \right]- \frac{r}{\theta} $$ $$= \left( \frac{\theta}{1-\theta} \right)^r r(r+1) \frac{1}{1-\theta} \frac{(1-\theta)^{r+1}}{\theta^{r+2}} - \frac{r}{\theta} = r\frac{r+1 - \theta}{\theta^2} $$ From which $$\sigma^2 = E(k^2;r) - \left( E(k;r) \right)^2= r \frac{1-\theta}{\theta^2}$$

which was to be shown.

Answer 3

In this answer the expectations are found without making use of the unfriendly distribution.

First we will solve it for $r=1$.

Let $T$ denote the event that the first toss produces a tail and let $H$ denote the event that the first toss produces a head. Then for $k=1,2,\dots$:

$$\mathbb{E}W^{k}=\mathbb{E}\left(W^{k}\mid T\right)P\left(T\right)+\mathbb{E}\left(W^{k}\mid H\right)P\left(H\right)=\mathbb{E}\left(1+W\right)^{k}\left(1-\theta\right)$$

Substituting $k=1,2$ we find two equalities in $\mathbb E(W)$ and $\mathbb E(W^2)$ that are not difficult to solve.

We find: $$\mathbb{E}W=\frac{1-\theta}{\theta}$$ and: $$\mathbb{E}W^{2}=\frac{\left(2-\theta\right)\left(1-\theta\right)}{\theta^{2}}$$ Then: $$\operatorname{Var}W=\mathbb E(W^2)-(\mathbb EW)^2=\frac{1-\theta}{\theta^2}$$

Now the general case: $W=W_{1}+\cdots+W_{r}$ where the $W_{i}$ are iid, leading to: $$\mathbb{E}W=\mathbb{E}W_{1}+\cdots+\mathbb{E}W_{r}=r\mathbb{E}W_{1}=\frac{r(1-\theta)}{\theta}$$ and: $$\operatorname{Var}W=\operatorname{Var}W_{1}+\cdots+\operatorname{Var}W_{r}=r\operatorname{Var}W_{1}=\frac{r(1-\theta)}{\theta^2}$$

Answer 4

$W$ is the sum of $r$ independent random variables $W_1,\ldots,W_r$ each of which counts the number of tails before one gets one head. Therefore the mean and variance are respectively $r$ times the mean and variance of any of those independent random variables.

So $\Pr(W_1=k)=(1-\theta)^k\theta$ for $k=0,1,2,3,\ldots\,{}$. One way to find $\operatorname{E}(W_1)$ is just applying the definition: \begin{align} \operatorname{E}(W_1) & = \sum_{k=0}^\infty k\Pr(W=k) = \sum_{k=0}^\infty k(1-\theta)^k\theta \\ & = \sum_{k=0}^\infty \sum_{j=1}^k (1-\theta)^k \theta \quad (\text{This last sum has $0$ terms when $k=0$.}) \\ & = \sum_{j=1}^\infty \sum_{k=j}^\infty (1-\theta)^k \theta \\ & = \sum_{j=1}^\infty \frac{(1-\theta)^j\theta}{1-(1-\theta)} = \sum_{j=1}^\infty (1-\theta)^j = \frac {1-\theta} \theta. \end{align}

To find $\operatorname{E}(W_1^2-W_1)$ is algebraically a bit simpler than to find $\operatorname{E}(W_1^2)$ directcly: \begin{align} \operatorname{E}(W_1^2-W_1) & = \sum_{k=0}^\infty k(k-1)(1-\theta)^k\theta \\ & = \theta(1-\theta)^2\sum_{k=2}^\infty k(k-1)(1-\theta)^{k-2} \\ & = \theta(1-\theta)^2\sum_{k=2}^\infty k(k-1)\eta^{k-2} \\ & = \theta(1-\theta)^2\sum_{k=2}^\infty \frac{d^2}{d\eta^2} k\eta^{k-2} \\ & = \theta(1-\theta)^2 \frac{d^2}{d\eta^2} \sum_{k=2}^\infty \eta^k \\ & = \cdots \end{align} The last series is geoemtric. Add it up; do the differentiation; then replace $\eta$ with $1-\theta$. Then by using the known value of $\operatorname{E}(W_1)$, you can find $\operatorname{E}(W_1^2)$ and then $\operatorname{var}(W_1)=\operatorname{E}(W_1^2)-(\operatorname{E}(W_1))^2$.

Two issues:

• That $\sum\dfrac{d}{d\eta}\cdots=\dfrac{d}{d\eta}\sum\cdots$ is easily shown when the sum has finitely many terms. That it works with infinitely many terms is problematic. A well-known theorem that's not that hard to prove says it does work when the sum is a power series and one is in the interior of the circle of convergence.

• Perhaps it is inelegant for me to use two very different methods for the two expected values. I have a preference (on this particular occasion of this particular posted question) for the first method because it avoids the issue in the first bullet point above. (The second method might be somewhat more generally applicable and thus preferable in some other contexts.) My excuse is that there are only 24 hours in a day and I must attend to some other things.