This is a practical question, I'm not sure if it's on-topic here. So sorry if it's not.
There is a competition where judges decide the score competitors by summing their performance mark in several aspects, and different competitors might be graded by different sets of judges.
E.g. There are competitors A,B and Judges 1~3. A is graded by J1 and J2, B is graded by J1 and J3.
But the judges is not professionally trained so the cardinal value of the score might not be comparable with each other. For example, J1 might prefer to give a score between 2~4, J2 between 1~5, and J3 between 3~5. How should I adjust the scores so that the average score reflect the truth strength of the competitors?
My thoughts:
(1) Naive ranking: If J1 grades $n_1$ competitors, then the $k$-th highest scorer gets $k/n_1$ points
(2) Normalized ranking: Normalize $\{k/n_1\}$.
(3) Normalized score: Normalize the score given by each judge.
But they have their advantages and disadvantages. So I would like to ask, what kind of method is most frequently used (I didn't go for "optimal", because it's hard to define which method is better)? Thanks!
