1

I am reading about the False Discovery Rate and Bonferroni and I found in various places that under H0 (true null) the distribution of p-values (for the test that rejects H0) should be uniform.

Can someone tell me why it would be the case?

Thanks!

ℕʘʘḆḽḘ
  • 113

2 Answers2

2

Consider a one-sided test that rejects if the statistic $X$ is too small.

If under the null hypothesis $X$ follows a distribution with CDF $F_X$, then the p-value is $F_X(X)$, the probability of obtaining an even smaller value. You can check that plugging in a random variable into its own CDF leads to a uniform random variable:

$$P(F_X(X) \le t) = P(X \le F_X^{-1}(t)) = F_X(F_X^{-1}(t)) = t, \qquad t \in [0,1].$$ This observation is used in inverse transform sampling.

angryavian
  • 89,882
  • thanks but the P value is usually 1 - F(X) so the probability of getting a test statistic as extreme as the one obtained. How does that change your proof? – ℕʘʘḆḽḘ Apr 30 '21 at 01:30
  • 1
    @ℕʘʘḆḽḘ If $S(x) = 1- F(x)$ is the survival function, then $P(S(X) \le t) = P(X > S^{-1}(t)) = S(S^{-1}(t))=t$. The inequality changes direction in the first step because $S$ is a nonincreasing function. – angryavian Apr 30 '21 at 01:41
1

Surprisingly, this isn't actually a duplicate on this site. It is however well answered in more detail on CrossValidated: https://stats.stackexchange.com/questions/435833/the-distribution-of-the-p-values-under-the-null-hypothesis-is-uniform0-1 .

The p-value is defined to be the probability of obtaining test results at least as extreme as those actually observed (when H0 is true). Thus, about 5% of the time, the calculated statistic should be < 5%. And this is holds for all values: about x fraction of the time, the p-value should be under x. The CDF is linear between 0 and 1, which is just the uniform distribution on (0,1).

wnoise
  • 2,281