Real-life question: How to determine the winner of a quiz

Question

Imagine there's a quiz on the internet intended for a wide audience. It contains a (unlimited) number of questions, all of them with yes/no answers. A person gets one random question and must answer it, after that he can get another one. He can continue answering any number of questions he wants. So the only data you have is:

Peter: 37 out of 45 questions correct
Mary: 42 out of 190 questions correct
John: 0 out of 127 questions correct
(etc.)

Now if you want to make a list of winners, how would you sort them the clever way?

• If you sort the people simply by the percentage of person's correct answers then the easy way to win is to correctly answer just one question (or two, three, ...).
• If you sort the people simply by the number of person's correct answers then an easy way to win is to spend a loong evening answering (or merely guessing) the questions. Then Mary (42 out of 190 correct) wins over Peter (37 out of 45 correct) even if Peter is significantly better.

So is there any sophisticated way how to choose a proper winner based on having just this type of data? Any statistical method that puts the best person higher than all these easy-winners?

Answer 1

You are trying to combine two different measures into a single rating: correct answer rate, and total number of answers, with the reasoning being that if two people have the same percentage correct, then it is less likely that the person with more overall questions attempted would have received that percentage by chance, hence you are ranking by both the raw percentage and the uncertainty in using that percentage to rank people accoring to the true, yet unknown, answer rate (assuming they answered an infinite number of questions).

A convenient way to come up with an "uncertainty"-weighted score is to use the Laplace estimator of the "correct guessing rate" for each person by teating their guesses as tosses of a fair coin with a 50% chance of getting "heads". This boils down to the following for a particular person, i:

$\hat p_{correct,i} =\frac{\#_{correct,i}+1}{n+2}$

What this estimator does is initially assume everyone is just guessing (when n=0) and hence all players have p=1/2. However, as data accumulates, the estimate approaches the observed fraction of correct answers. If two players, A and B, have the same observed percentage correct (i.e., $\%_A=\%_B$) but $n_A>n_B$, then $\hat p_A>\hat p_B$, satisfying your requirement that total guesses be taken into account.

EDIT 1: Expanding to other priors

The Laplace estimator is a special case of Bayesian estimation using a Beta-Binomial model. In the Laplace case, the prior is assumed to be a uniform density on [0,1]. This is a very diffuse prior, in that is says that 10% of the population would answer at least 90% of the answers correct, and 10% would get no more than 10% correct.

This is probably an unrealistic assumption about the knowledge of the general population. Most likley, people will end up guessing on a lot of questions, so we'd expect the prior to be more concentrated aroun 50%. To accomodate different assumptions about the general population's knowledge, you can use the Beta-Binomial model I linked to, with different parameters for the beta prior on [0,1] to represent your guess/estimate/assumptions about the population's general knowldge about your questions. The more concentrated you make your beta around 50%, the more correct/incorrect answers are required to shift the estimate for person i's "long run correct answer rate" away from 50%. Right now, its pretty easy to do so.

This type of estimate is called a "shrinkge estimate", the version for the beta-binomial is also in the link is sent you, here, where $\theta_i$ is the estimated long run fraction correct for person i.

Try out different beta priors to see what makes sense. Also consider how knowledgeable your population will be about the subject matter of your questions. The more knowlegeable, the more you should bias the beta towards 1. The less knowledgeable, the more it should concentrate around 50%. There really is no situation where you would bias downwards unless you purposefully put in tricky questions.

Answer 2

Here's a more naive, elementary answer that might be useful. Note that I admit to being bad at both probability and combinatorics, so you probably shouldn't accept any of this without double checking the calculations.

The first issue is to define what you mean by "winner". Perhaps we can assume that, as a person answers more and more questions, their correct answer ratio will tend towards some limiting value. Specifically, we might assume that a given person knows the answer to a given subset of the questions, and doesn't know the answer to another subset of the questions. If $A$ is a person, we could write $\text K(A)$ for the number of questions that $A$ knows the answer to.

You can't know what $\text K(A)$ is for sure, but for any given $n$, you can determine the probability that $\text K(A)=n$. Let $Q$ be the overall number of questions (You say there's an "unlimited" number of questions, but surely in practice it would be finite, right? Even if very large). If you measure that $A$ got $q_r$ questions right and $q_w$ questions wrong, we want the probability that a person with those numbers would know $n$ questions. There are $\binom{Q}{n}$ ways to be a person that knows $n$ questions, and if my reasoning is correct (it very well may not be, because I'm notoriously bad at combinatorics), $\binom{Q-q_r-q_w}{n-q_r}$ of those would yield the observed results. So:

$$\text P\left(\text K(A)=n \mid q_r, q_w\right)=\frac{\binom{Q-q_r-q_w}{n-q_r}}{\binom{Q}{n}}$$

So, the probability that a person $A$ knows more than a person $B$ is going to be equal to something like this:

$$\sum_{Q>n>m}\text P(\text K(A)=n)\text P(\text K(B)=m)$$

Where the probabilities in question are calculated according to above formula.

So now, among all the people who took the quiz, you have a weighted directed graph giving the probability that each person knows more than each other person. For each person you can now calculate the probability that they know more than anyone (in theory, at least - not sure what the formula would be, because I doubt that knowing more than two different people are independent events), and work out who most likely has the most knowledge.

If you're committed to considering the question set as infinite, you would have to make the assumption that as a person answers more questions, their ratio will tend towards some limiting value which is uniquely defined for each person. You then need to use the observed data to work out the probability that a person has a given "true" ratio, the probability that one person has a higher ratio than a given other person, and figure out who most likely has the highest ratio of all.

Answer 3

I suppose the problem is if you have an infinite list of questions, any participant could scroll down the list indefinitely and only answer those questions they knew the answer to. But they could also do this for arbitrarily large lengths of time thus obtaining both a very large number of questions correct, and a 100% consistency rate. Since anyone can do this regardless of skill, I think the test itself is flawed. To fix this you could perhaps add a time limit.