1

I am setting up a Fermi Quiz. A quiz where you have to estimate the answers to complicated questions, your answers may be out by orders of magnitude.

A question may be:

"How many piano tuners are there in Chicago?"

The correct answer being around 290, however any answer between 140 and 580 is considered quite close.

What would be the best way to calculate the score for such a quiz.

user288447
  • 248
  • You make the rules here. Why is 290-150 and 290+290 considered quite close? How about one mark if in the range, zero marks otherwise? I mean someone might come up with 289 but 290 is still quite far away. You could also do something like giving them something like $1-\frac{|P_0-P_1|}{|P_0|}$ marks --- where $P_0$ is the correct answer and $P_1$ is their guess. So that, for example, 290 would give you one mark and 280 would give 0.9655 marks – JP McCarthy Sep 30 '14 at 15:00
  • If I do the $1-\frac{P_0 - P_1}{P_0}$ then a person who is "only double" the correct answer will get zero marks. – user288447 Sep 30 '14 at 15:17
  • Yes correct I am not gone on that either. You could take logs. E.g. $\log 799\approx 6.683$ and $\log 290\approx 5.67$... – JP McCarthy Sep 30 '14 at 15:18
  • 1
    Yeah I am thinking a good answer will have something to do with logs, but not sure how to best implement it. – user288447 Sep 30 '14 at 15:20

1 Answers1

1

I have been thinking about this a lot, because i wanted to make something similar.

In my opinion the best method for scoring is based on the answers of the other participants in the quiz.

You get 100 points if the answer is exactly the right answer, otherwise you get 100 points minus the percentage of participants whose answer is better than yours.

An answer $A$ is better than yours if there are less answers between the correct answer and $A$ than between your answer and the correct answer.

This might seem difficult to calculate, but it is quite easy if you have the distribution of answers available.

Example: if the correct answer is $100$, and the set of answers is ${1,15,50,70,80,95,99,101,110,300,1000}$ the scores would be:

$1:0$ points

$15:10$ points

$50:20$ points

$70:40$ points

$80:60$ points

$95:80$ points

$99:100$ points

$101:100$ points

$110:80$ points

$300:60$ points

$1000:40$ points

Ward Beullens
  • 1,903
  • A problem arises where everyone is very poor at estimating... for example, ${1,2009,799}$ estimates of 290 and here the person who guessed one will get one mark, 799 will get 0.50 and 2009 only 0 --- fair enough with 2009 but is 799 really that much worse than 1? You have to quantify "better answer", I little better perhaps. – JP McCarthy Sep 30 '14 at 15:03
  • Actually in this case, the guess 1 and 799 both get 100 points, and 2009 gets 0 points – Ward Beullens Sep 30 '14 at 15:06
  • Why is that? How do you say that 799 and 1 are both as "good" as each other? – JP McCarthy Sep 30 '14 at 15:07
  • They both are 'next to' the right answer, but 2009 is seperated from the right answer by 799. – Ward Beullens Sep 30 '14 at 15:08
  • But obviously this would work better if there are more participants. In general I would say that if all participants make very poor estimates it is justifiable for a poor (not very poor) estimate to get a high score. – Ward Beullens Sep 30 '14 at 15:10