0

There are a large number multiple testing p-value correction methods. e.g.:

   bonferroni : one-step correction
   sidak : one-step correction
   holm-sidak : step down method using Sidak adjustments
   holm : step-down method using Bonferroni adjustments
   simes-hochberg : step-up method (independent)
   hommel : closed method based on Simes tests (non-negative)
   fdr_bh : Benjamini/Hochberg (non-negative)
   fdr_by : Benjamini/Yekutieli (negative)
   fdr_tsbh : two stage fdr correction (non-negative)
   fdr_tsbky : two stage fdr correction (non-negative)

(based on https://www.statsmodels.org/dev/generated/statsmodels.stats.multitest.multipletests.html)

I have found a lot of pages that explain the methods individually (and why corrections are needed) but I have not found an overview of when to use which method e.g. a comparison table or even better a decision flow diagram as it exists for machine learning methods.

Any ideas? How do I decide which multiple testing correction I should apply?

Disclose: I already posted this in data science but even with a bounty no useful answer was given. Maybe here is more appropriate?

lordy
  • 101
  • The reason why you aren't getting an answer is because the answer depends on the experimental design and the hypothesis you are testing. If you don't understand why this is the case, you should not be performing multiple hypothesis tests because you will very likely make an incorrect inference if you do. – heropup Feb 03 '21 at 09:05
  • "The reason why you aren't getting an answer is because the answer depends on the experimental design and the hypothesis you are testing." Exactly. But still there needs to be some generally accepted way which design and hypothesis needs which test. – lordy Feb 03 '21 at 12:26

0 Answers0