How biased is this biased coin

Question

Suppose that we have a coin that we suspect is biased, but that we don't know precisely how biased it is: all we know is that its probability p of landing heads is some fixed value between .4 and .6, inclusive.

We flip the coin 100 times, and it lands heads 70 times. I'm curious how to find the probability that .4 ≤ p ≤ .55.

My approach was to find

$$\frac{\int_{.4}^{.55}\binom{100}{70}p^{70}(1-p)^{30} dp}{\int_{.4}^{.6}\binom{100}{70}p^{70}(1-p)^{30} dp} ≈ .057$$

but this seems too simplistic. Where am I going wrong?

EDIT: Apologies, I meant to say that p is uniformly distributed on [.4, .6], though I'm now curious how we would solve if we knew that p is normally distributed on [.4, .6].

Answer 1

This is a question that is naturally suited to a Bayesian approach. Suppose our prior belief about the true probability of heads $p$ is modeled by some distribution $f(p)$. Then, we conduct the experiment of flipping the coin $n$ times and observing the number $X$ of heads, which is assumed to follow a binomial distribution, specifically $$X \mid p \sim \operatorname{Binomial}(n,p).$$ Thus $$\Pr[X = x \mid p] \propto p^x (1-p)^{n-x}$$ represents a likelihood function $L(x \mid p)$ for the sample, and the posterior distribution of our belief about the parameter $p$ is given by Bayes' theorem $$f(p \mid x) \propto L(x \mid p) f(p).$$ For a Bernoulli/binomial likelihood, the choice of prior distribution that gives a posterior in the same parametric family happens to be a beta distribution: i.e., if $p \sim \operatorname{Beta}(a,b)$ for suitable hyperparameters $a, b$, the posterior $p \mid x \sim \operatorname{Beta}(a^*, b^*)$, for new posterior hyperparameters $a^*, b^*$; specifically, $$a^* = a + x, \quad b^* = b + n - x.$$ (The proof of this is left as an exercise for the reader.) However, when the prior is not beta distributed, the posterior may not be either. We explore this calculation for the choice of prior $$p \sim \operatorname{Uniform}(0.4, 0.6).$$ We calculate: $$f(p \mid x = 70) \propto \begin{cases} p^{70} (1-p)^{30}, & p \in [0.4, 0.6] \\ 0, & \text{otherwise}. \end{cases}$$ Note the subtlety: because the prior for $p$ was confined to $[0.4, 0.6]$, the posterior is also necessarily confined to this range, even if the sample proportion $\hat p = x/n$ is not in this range, because $$f(p) = \begin{cases} 5, & p \in [0.4, 0.6] \\ 0, & \text{otherwise}. \end{cases}$$ Consequently, the posterior density must be $$f(p \mid x = 70) = \frac{p^{70}(1-p)^{30}}{\int_{p=0.4}^{0.6} p^{70}(1-p)^{30} \, dp},$$ and the probability that $p \in [0.4, 0.55]$ is simply $$\Pr[0.4 \le p \le 0.55] = \frac{\int_{p=0.4}^{0.55} p^{70}(1-p)^{30} \, dp}{\int_{p=0.4}^{0.6} p^{70}(1-p)^{30} \, dp} \approx 0.0571106,$$ as claimed.

Now, if we use a different prior for $p$, the posterior distribution and desired probability will also be different. As it is nonsensical to say "normally distributed on $[0.4,0.6]$," let us use a suitable beta prior that "looks roughly normal" on this interval. Clearly, the mode should be at $0.5$, consequently such a beta prior must have $a = b$. As to the choice of common hyperparameter, if we assume that roughly $95\%$ of the probability density should be in $[0.4,0.6]$, then numerical approximation gives the choice $a \approx 47.2998$, so let's choose $a = b = 48$. Then, as this prior is conjugate, we simply have $$p \mid x \sim \operatorname{Beta}(a^* = 48 + 70 = 118, b^* = 48 + 30 = 78),$$ and it follows that $$\Pr[0.4 \le p \le 0.55] \approx 0.0695632.$$

Answer 2

Your integration calculation looks correct.

As an illustration of the issue, if $p=0.6$, the probability of seeing $70$ out of $100$ heads is about $0.0100075$.

By contrast, if $p=0.55$, the probability of seeing $70$ out of $100$ heads is about $0.0007757151$, less than a twelfth of the earlier probability, and the probability would be even smaller for smaller $p$.

So if you really believe your statement "all we know is that its probability $p$ of landing heads is some fixed value between $0.4$ and $0.6$" then seeing $70$ out of $100$ heads should push your belief strongly towards thinking $p$ is likely to be close to $0.6$.

This is what your Bayesian calculation with a prior uniform distribution on $[0.4,0.6]$ does. Indeed you would find that there is a posterior probability above $0.5$ that $0.586 \le p \le 0.6$.

Any prior distribution which gives a reasonable probability of $p$ being close to $0.6$ will, after seeing $70$ out of $100$ heads, give a small posterior probability of being below $0.55$.

Answer 3

A good way of thinking about this is in terms of the following thought-experiment. We randomly select a large number of fake coins. Each coin has some random value of $p$ assigned to it, by drawing from a probability distribution $\rho(p)$. We flip each fake coin $100$ times. If the result is not $70$ heads, we discard it. How are the results that we are interested in (= $70$ times head) distributed over $p$? We can compute the Bayesian probability that $p$ came from the interval $(a,b)$ is given by:

$$P(a \le p \le b) = \frac {\int_{a}^{b} \rho(p)p^{70}(1-p)^{30}dp} {\int_{0}^{1} \rho(p)p^{70}(1-p)^{30}dp}$$

Note that I have omitted the factorials in denominator and numerator, since they do not contribute anything and simply cancel. You can substitute any a priori estimate for the probability distribution $\rho(p)$ and evaluate the above equation. The factor $p^{70}(1-p)^{30}$ is sharply peaked around the value $0.70$. As a consequence intervals $(a,b)$ that contain this value tend to have a high probability, whereas intervals without it tend to have a low probability.