Testing significance of patterns of results

Question

I'm a high school English teacher conducting an independent study, and I'm a total novice to statistical analysis, so please forgive me if I mischaracterize anything. I have gathered pretest and posttest data about student motivation from three groups of students: one that received normal grades over a six-week period (Group 3), one that received delayed performance-contingent grades and immediate completion-contingent grades (Group 2), and one that received no grades (Group 1). The tool that I used measures seven types of motivation, which I've coded as IMK, IMA, IMS, EMID, EMIJ, EMX, and AM. My hypothesis is that feedback in the form of narrative evaluations without immediate or salient multi-interval grades (i.e., Group 1 and Group 2) will lead to better motivational outcomes for high-school students in autonomy-supportive classrooms than forms of feedback associated with immediate and salient multi-interval grades (Group 3). "Better," in this case, means that IMK, IMA, IMS, and EMID will increase and EMIJ, EMX, and AM will decrease by the end of the six-week period for students in Group 1 and Group 2. My hypothesis for Group 3 is two-tailed; I think both grades and autonomy support will have an effect, but I don't want to make any predictions about what that effect will look like - my only prediction is that the motivational outcome will be worse.

I ran a bunch of paired t-tests to check the significance of the difference between pretest and posttest scores for each measured variable. Here's what I came up with:

https://i.stack.imgur.com/NPB82.png

Now, at a glance, you can see that the results seem to support my hypothesis. In Group 1 (Withheld), for instance, there was an increase in each of the things that I thought would increase, and there was a decrease in each of the things that I thought would decrease. It doesn't seem intuitively likely that this specific pattern happened randomly, even if the changes for each individual variable didn't pass the significance threshold. What sort of test could I do (ideally in R) to prove this?

Answer 1

Fair warning that there's a lot of information in this reply as I try to clarify the problem while providing an appropriate solution!

As I understand, you have quite a bit going on here so I'm going to try and break out all the information a bit. To start, you have either 4 or 7 outcomes (based on whether you predict composite scores or dimensional scores) which are all ratio variables.

• DVs: IMK, IMA, IMS, EMID, EMIJ, EMX, AM

You then have a single predictor which is a nominal factor. It is simply an assignment of group membership (1,2,3) but does not have numeric meaning in itself (i.e., 1 < 2). So, we have:

• IV: grading scheme (3 levels)

Finally, it seems you have three hypotheses:

• H1: There will be an effect of grading scheme on: a) internal motivation, b) autonomous motivation, c) controlled motivation, and d) amotivation.
• H2: Compared to Group 3, Group 1 will have a) higher internal motivation, b) lower autonomous motivation, c) lower controlled motivation, and d) lower amotivation.
• H3: Compared to Group 3, Group 2 will have a) higher internal motivation, b) lower autonomous motivation, c) lower controlled motivation, and d) lower amotivation.

Assuming that all this is a proper interpretation of what you're doing, I believe the following steps are what you should take.

• To test H1: you will want to run a one-way repeated-measures MANOVA (multiple DVs, single factor predictor). Necessary assumptions behind MANOVA linked here. Alternatively, you could skip ahead and just run individual one-way repeated-measures ANOVAs for each of your DVs to assess whether there are differences in group means. Necessary assumptions behind one-way repeated-measures ANOVA. This will give you an omnibus F that denotes whether grading scheme has an overall effect on motivation. Here is an R vignette on how to conduct repeated-measures MANOVAs in R
• To test H2 & H3: provided H1 is supported, you will want to perform follow-up one-way repeated-measures ANOVAs. Here is an R tutorial on how to do that. Provides, those are then significant, you will want to perform a post-hoc Tukey's honest significant difference (HSD) test using tukeyHSD() in R. This "basically" performs pairwise t-tests between the different groups but accounts for something called family-wise error. When you do a bunch of t-tests on a data set like this, you inflate the odds of detecting a false positive. Tukey's HSD accounts for this.

Hope this is helpful!