2

In my test I alter one parameter in three stages (p1

After plotting it, it is quite obvious that the parameter positively effects the outcome, because all 3 lines are above each other (no intersection).

However, when I do the 3 paired t-test (p1 vs p2, p1 vs. p3, p2 vs. p3), the p-value of the test between the lower and the middle line is smaller than between the lower and the upper line. Logically it should not be the case, since the gap is larger for the extreme cases, right?

I was wondering whether my t-test is wrong or whether the fact that the curves are not parallel and the upper line is fluctuating a bit more (but still never intersects even the middle line!) overrules the larger gap and therefore explains the worse p-value. (all p-values are very low, the bigger gap between the extremes is 1,70*E-23, the smaller gap between low and middle is 2,93*E-25)

Looking forward to hearing your expertise on the subject. :)

UKWildcats
  • 21
  • (1) The nature of your plot is not clear. (2) The P-value of a paired t test depends on the SD of the differences as well as the mean of the differences, so (if I understand it correctly) your 2nd paragraph is not surprising. (3) Each of the 3 t tests has it's own type I error ($\alpha$ level, signif. level). The type I error of the combination if the three t tests is difficult to quantify. (4) I guess you may have a more complex ANOVA design in the background. If so, all this might make more sense if the design were specified. – BruceET Aug 07 '16 at 01:41
  • Hi Bruce, Thanks for your reply. Yes, they all have their own type I error, it is really a simple paired t-test. The graph looks like this: https://s8.postimg.org/dzwhcglqd/DSC_0016.jpg From first look I would assume that conducting a paired t-test between p1 and p3 would yield the lowest p-falue (most significant), but in reality, comparing p2 and p3 yields the lowest value. I just wanted to check whether this can be explained by the standard deviation itself or whether it indicates that I did something wrong. – UKWildcats Aug 07 '16 at 09:52
  • Guess your answers are OK and it's the SD. Would need to see details to be sure. – BruceET Aug 07 '16 at 16:22

1 Answers1

1

Demonstration, using normal differences and paired t tests

Two sets of differences, each tested for population mean = 0 using paired t test. Sample size is 20 in each case.

 mean(d1); sd(d1)
 ## 1.9315            # smaller diff from 0
 ## 2.316931          # smaller SD
 t.test(d1)$p.value
 ## 0.00142518        # smaller p-value (more signif result)

 mean(d2); sd(d2)
 ## 2.108             # larger diff from 0
 ## 3.214422          # larger SD
 t.test(d2)$p.value
 ## 0.008539601       # larger p-value

Slanted red line connects sample means.

enter image description here

BruceET
  • 51,500