0

I am potentially doing multiple pairwise comparisons in this question and the cited answer warns me to adjust for multiple comparisons using for example Bonferroni. However, I am not sure what p-values to adjust by how much.

Szenario: I have a sack of fruites with Apples, Oranges, and Pears which can either be ripe or not.

When I am testing if any of the fruits differs significantly in the amount of ripeness from the other fruits, I am performing 3 tests and thus need to use $\alpha/3$ as the adjusted significance level. However, what if I now test a second attribute, i.e., sweetness? Is this a different test battery and I again need to adjust only to $\alpha/3$ or do I now perform 6 tests and thus have to adjust to $\alpha/6$?

Finally, I have a second sack of fruits with dates, tomatoes, and strawberries for which I also want to perform the same test battery. Does this mean I have the same adjustments again (either $\alpha/3$ or $\alpha/6$) or do those tests also count towards the test count and I have to up the overall adjustment to $\alpha/6$ or $\alpha/12$?

When adjusting $\alpha$ for multiple significance tests what tests count towards the adjustment denominator when using Bonferroni?

Sim
  • 208

1 Answers1

1

The Bonferroni method controls the familywise error rate (FWER), which is the probability of rejecting at least one true null hypothesis. Particularly, it uses a significance level of $\alpha/N$ for each test to ensure the FWER for $N$ tests is no more than $\alpha$. It doesn't matter how these tests relate to one another; if you care about the FWER among $N$ total tests, then you use $\alpha/N$.

However, it is also worth noting that the Bonferroni method is a very conservative approach in that it over-rejects null hypotheses and doesn't lead to enough "discovery." A more powerful procedure that still controls the FWER at level $\alpha$ is the Holm method.

You may also be interested in controlling something other than FWER, such as the false discovery rate. There are many other approaches to deal with the multiple testing problem.

Golden_Ratio
  • 12,591