Why is the statement false: If a test is rejected at significance level α, the probability that the null hypothesis is true equals α?

Question

Since the answer is false, so it just means the probability that null hypothesis is false equals α. But α is defined as the probability that null hypothesis is rejected when it is actually correct. I don't see any connections between these two statements. Could someone give me some explanations on this? Thanks!

Answer 1

The short answer is that the sampling distribution of the test statistic assumes the null hypothesis is true; therefore, if you reject $H_0$ at significance level $\alpha$, that simply means that if $H_0$ were true, the chance that you could have obtained the sample that you did (or something more extreme) is less than $\alpha$. It doesn't mean the probability that $H_0$ is true is $\alpha$, because the test itself is predicated on the assumption that $H_0$ was true. $\alpha$ is the probability of error due to having observed an exceptional sample, not the chance that $H_0$ is true, because in the frequentist viewpoint, the true state of affairs is that exactly one of $H_0$ or $H_a$ is true.

Suppose I give you a coin, for which I know its probability of landing heads is $p$, the true population parameter. But I don't tell you. You can toss it as many times as you wish, but if I don't tell you what $p$ is, you can never actually know. Suppose you are interested in testing $H_0 : p = 0.5$ versus $H_a = p \ne 0.5$ at $\alpha = 0.01$; i.e., you are interested in determining if the coin is fair at a $99\%$ confidence level. You toss the coin $n = 20$ times and obtain $X = 15$ heads. Your resulting estimate for the parameter $p$ is $\hat p = 15/20 = 0.75$. Informally, you might think this suggests the coin is biased; after all, $0.75$ seems quite different from $0.5$. But is this so unusual that, if the coin IS in fact fair, the chance you could've gotten such an extreme result is no more than $1\%$? In other words, assuming that the coin is fair, how likely is it to have obtained $15$ or more heads, or $5$ or fewer tails? This probability is a sum of binomial probabilities and is $\frac{5425}{131072} \approx 0.041$. So, while it is a rare event, it isn't sufficiently rare for you to be $99\%$ confident you have correctly asserted the coin is unfair. Consequently, you fail to reject $H_0$ at the $\alpha = 0.01$ level. It doesn't mean that the coin's probability of being fair is $1\%$. That obviously makes no sense, since I could have done the same test at $\alpha = 0.00001$ and again failed to reject $H_0$. Even the $p$-value of $0.041$ is not the probability that the coin is fair, because $$p {\rm -value} = \Pr[X \le 5 \cup X \ge 15 \mid p = 0.5].$$

Furthermore, the true state of reality is that the coin is either fair or unfair. It is not a random event. There is no probability attached to either of these possibilities, because I know for certain whether it is or not. That's the frequentist viewpoint, as I said earlier.

Answer 2

Think about what the significance level $\alpha$ is. If we have some test statistic $T$, then the significance level $\alpha$ is just some criterion that we are trying to satisfy to make some form of an objective opinion about whether a model is appropriate or not (not whether a model is true or not). The importance about the level $\alpha$ is that if we have a test statistic, we reject the null hypothesis if $\text{P}(T\text{ occurs under $H_0$}) \leq \alpha$. This statement says that "If the probability that our observed test statistic could occur under $H_0$ is less than or equal to $\alpha$, then we'll reject the claim that $H_0$ is appropriate". Note that this is not the same as saying that the probability that the null hypothesis is true with probability $\alpha$. One talks about the observed Test statistic we have, whilst another talks about the true model, which is what we (statisticians) attempt to model as appropriately as possible, and use hypothesis testing as a diagnostic tool.

Answer 3

A rejection of the null hypothesis at significance level $\alpha$ means that if the null hypothesis is true, the probability of observing a test statistic as extreme as that observed is less than $\alpha$. The probability that the null hypothesis is true given the observation of the test statistic is another object, and will involve the specification of a prior over the truth of the null hypothesis.

Basically it boils down to P(data | hypothesis) != P(hypothesis | data).