Note that I updated question because initial example grades did not satisfy the overall average that followed from the numbers given in the article.
A newspaper artticle in Norway made me curious about this one. https://translate.google.com/translate?sl=no&tl=en&js=y&prev=_t&hl=en&ie=UTF-8&u=http%3A%2F%2Fwww.dn.no%2Ftalent%2F2015%2F06%2F30%2F1025%2FEksamen%2Fda-elevene-sorterte-eksamenskarakterene-alfabetisk-etter-navn-gikk-det-opp-for-dem&edit-text=
In a school class there were 25 students who had the same exam, graded from 1 to 6. After the the exams had been graded a lot of the students felt something was wrong. So one them collected the names and grades of all the students and put them into an excel sheet. When he sorted the names alphabetically he noticed that the first 13 had on average better grades than the last 12. In fact, the average of the first 12 was 3.4 and the average of the last 13 was 1.8.
When confronting the school it was confirmed that the class had been split in half, by name, and one pair of evaluators graded the first 13 students, and a different pair of evaluators graded the last 12 students. (there are always 2 persons evaluating each exam submitted)
Now we are debating the probablity that this difference in averages occured randomly because stuff just happens, and the probablity that the different evaluators was the cause of this.
Unfortunately I don't know the grades. But suppose the grades i the entire class were:
1,1,1,1,1,1,1
2,2,2,2,2,2
3,3,3,3,3,3
4,4,4
5,5
6
Group A:
1
2,2
3,3,3,3
4,4,4
5,5
6
Group B:
1,1,1,1,1,1
2,2,2
3,3
4
These numbers gives close to the averages mentioned. (but the distribution feels kinds of wrong, but overall average was very low...)
I'm not sure of the math here, so I ran a simulation and found that there is about 16% chance that the difference in averages between 2 randomly created groups of sizes 12 and 13 is 3.4-1.8=1.6 or larger.
16% is not that small a number. This will happen many times each year in classes at many schools. But in this case that we also know that there were different pairs of evaluators in the 2 groups, how should I interpret this? Is it correct to say it is 84% chance that the large difference was beacuse of different evaluators? Are there good reasons for the group with the worst grades to demand new evaluations?
I think the 16% number is correct, but doing the math would be better than running a sim. So some pointers on how to do that would also be appreciated.
