3

I know that the p-value is the probability that the measurement takes a value that is at least as unlikely under the null hypothesis as the observed value. Furthermore, the null hypothesis is rejected if and only if the p-value of the observation is smaller than the signification level.

My question is: Is the p-value then the probability that your null hypothesis is true? Why (not)?

Edit: I do not think it is, but I can not properly explain why.

  • @SimonS What convention? Also this site says no (but doesnt explain why): http://www.dartmouth.edu/~matc/X10/Show.htm –  Oct 04 '15 at 21:41
  • @SimonS Why did you delete your comment? –  Oct 04 '15 at 21:51
  • 1
    Your own question gives the answer; your first sentence (correctly) gives the condition "under the null hypothesis". Your subsequent question omits that condition; it's not true because of the omission. [NB There are numerous related questions on stats.SE; a good answer which at least partly addresses issues related to your question is here] – Glen_b Oct 05 '15 at 06:06

1 Answers1

1

No and this is a very common misconception!!!

First of all, let's write $H=1$ if the alternative is true and $H=0$ if the null is true. You are asking whether the p-value is equal to $\Pr[H=0]$.

First observation: From a frequentist point of view, your question does not even make sense, because either your hypothesis is true or it is not (i.e. $\Pr[H=0]=0$ or $\Pr[H=0]=1$).

But also if you relax this a bit and take a more Bayesian point of view, so that it could be the case $0<\Pr[H=0]<1$, then the question still does not make sense, because the p-value is a random variable, while the former probability is just a constant!!!

In particular let's look again at your definition of a p-value:

"the p-value is the probability that the measurement takes a value that is at least as unlikely under the null hypothesis as the observed value"

So based on this definition (and simplifying things a bit to e.g. to one-sided tests) you observe a random variable $X$ and then the p-value is just $P=1-F_0(X)$, where $F_0$ is the distribution of $X$ under the null hypothesis. Again the key point to observe here is that the p-value is random, because $X$ is random, so it can in principle (randomly) take any values between 0 and 1. Therefore it cannot be equal to a constant ($\Pr[H=0]$)!

The other important insight is that at least for continuous $F_0$, it will hold that $P \sim U[0,1]$ under the null hypothesis. For powerful tests the alternative distributions of the p-value will be stochastically a lot smaller than uniform, i.e. $Pr[P \leq t \mid H=1] > t$. Thus if you observe a very small p-value it is a lot more likely to have come from the alternative than from a uniform distribution. Again this can be made more precise in a Bayesian framework but also explains things from a frequentist point of view.

Still the most important thing to always remember is that the p-value is a random variable.

air
  • 2,812