2

I want to generate $n-1$ wrong answers $a_2,\ldots,a_n$ for a multiple choice test given a correct number $a$. The problem is that I don't want the user to be able to guess the correct option with an accuracy of more than $\frac{1}{n}$ if the user doesn't know the correct number. Is that possible and if yes, how?

Ideas

Always draw from the same interval

Ignoring the correct value $a$, just always choose $a_i$ randomly from $[$lower bound, upper bound$]$, e.g. [0,10000]. The problem is that if I ask for e.g. the year of the french revolution ($a=1789$), the user knows that it cannot be e.g. in year 2345. Even if I set the upper bound to the current year, giving the years 110, 720 and 2001 as wrong options will be too easy. Even if the answer isn't a year, a smaller value will be more probable in general.

Draw numbers a bit smaller, larger or around the correct number

If I generate slightly smaller numbers (lets say from the interval $[0.9 \cdot a,a)$) than the correct answer, the user just always chooses the largest one. If I always generate slightly larger ones, $(a,1.1\cdot a]$, the smallest one is always correct. And if I generate numbers around the number, $[0.9 \cdot a,1.1 \cdot a]$, the one in the middle will be more often correct than the others.

Two Step Process

I could first choose a method randomly, 1/3rd of the time I choose smaller, 1/3rd of the time larger and 1/3rd of the time "around" the number. Now I can still break this because if I see that $\frac{\max(a_i)}{min(a_i)} > 1.1$, then I know the last option was chosen and I can choose the middle one. If I change the "around" interval to $[0.95\cdot a,1.05\cdot a]$, the quotient could still be slightly over 1.1. I could use $[0.95\cdot a,0.95\cdot1.1 = 1.045 \cdot a]$ but now $a$ isn't in the center of the interval any more which can be exploited with a uniform draw of random numbers.

Is there any way to create random numbers similar to the correct numbers that don't give any information about it? Maybe I need to use a random number generator that doesn't have a uniform probability in the interval? Or should this exploitability just be factored into the scoring?

P.S.: I guess $0.9\cdot a$ should be $\frac{a}{1.1}$ so that the quotient is the same.

Konrad Höffner
  • 141
  • 5
  • 2
    I think this is potentially interesting but not at all well specified. You suggest that the test taker has some information about the correct answer ($2001$ can't be right for the French revolution.) Until you specify that kind of knowledge you can't quantify how much information your wrong answers convey. What precisely might "similar to the correct number" mean? In multiple choice questions wrong choices are often carefully chosen to be the results of wrong methods. – Ethan Bolker Apr 11 '19 at 12:18
  • @EthanBolker: I will try to specify it better, but this is hard to do because I want to abstract from a practical problem. Would it help to assume that the test taker knows that the number is inside some interval around the number, e.g. [0.5number,2number] but not where exactly it is in that interval? The result of wrong methods is a great idea, but unfortunately not applicable in my case because I need to generate the answers automatically. – Konrad Höffner Apr 11 '19 at 12:41
  • How much control do you have on the correct answer? If you have maths or physics problems, you can also adjust the problem so that the correct answer is say an integer less than 100. In this case you can first define a range of possible answers, then adjust the problem so that the correct answer is in that range and draw wrong answers from this range as well. If on the other hand you have say history problems where the correct answer is fixed your choice of wrong answers decides how hard the question will be. – quarague Apr 11 '19 at 12:54
  • @quarague: The answer is fixed, as the query is generated automatically using a database of facts. – Konrad Höffner Apr 11 '19 at 13:04
  • 1
    If the question are for some fixed real life facts, you need to pick a suitable range/class of possible answers. This has to depend on the particular fact and also has a huge influence on how hard the questions will be. For the French revolution example: if the wrong answer lie in 1 to 2000 it will be an easy question, if all the wrong answers are between 1750 und 1800 it will be a lot harder. That is a choice for the question poser, there is no maths solution to that. – quarague Apr 11 '19 at 13:14
  • 2
    This question should go to an educators' site, not a math site. They will be familiar with the copious literature on the topic. – Gerry Myerson Apr 11 '19 at 13:34
  • @GerryMyerson: I will look at educator's sites but I am not sure that would be a better fit as those seem to suggest thinking processes for humans but I want to generate the wrong answers automatically from a database using a formula. – Konrad Höffner Apr 11 '19 at 13:38
  • 1
    I repeat: there is a copious literature on the subject of constructing multiple choice questions, and the people familiar with that literature should be at the educators' sites. – Gerry Myerson Apr 11 '19 at 13:44
  • 1
    https://matheducators.stackexchange.com/questions/13058/how-to-design-multiple-choice-questions – user7610 Apr 12 '19 at 15:20

2 Answers2

1

Here are some thoughts about an abstract version of your question.

Assume a finite set of $A$ of possible answers (your questoin suggests that will be an interval of integers). Randomly choose $n-1$ elements from the set of wrong answers. Then provide as choices the correct answer and your wrong answers (place the correct answer at random).

If the test taker knows the answer they will choose it. If they have no information they will guess and succeed with probability $1/n$. If they have some information about the correct answer then they will be able to increase their probability of guessing correctly to some value greater than $1/n$. You can't avoid that unless you know more about how they are thinking.

If you knew their personal probability distribution over the set $A$ of possible answers (something between uniform and certainty) I think you could design a random set of wrong answers that provided minimal information in some sense. Quantifying that strikes me as an interesting and probably difficult problem. When they are certain and correct nothing you do will fool them. If they are certain and incorrect in a way you know should you allow their wrong choice to be one you include along with the correct answer? What would your best strategy be if they thought two of the possible answers were equally likely to be correct?

I suspect this is the kind of game cryptanalysts play,

Ethan Bolker
  • 95,224
  • 7
  • 108
  • 199
  • I cannot avoid them having a higher chance of choosing correctly if they have some information but I would like minimize that chance. For example if they know the century of the French revolution, and the wrong answers lie in other centuries, the player will always guess correctly, which seems a bit high for me. I don't have information about the mind of the players, but I will try to make the numbers as close as possible to the correct one, I assume this will minimize the information given. – Konrad Höffner Apr 15 '19 at 10:06
1

Let me try to think of this as a classification problem. The student has a set of options and needs to decide whether the option is a correct answer or not. A probabilistic view is also possible, where the student gives odds on each of the answers.

Each student has different abilities to perform the classification. On one end of the spectrum, student knows the correct answer even before looking at the options. On the other end, student thinks all answers are equally likely, because the student knows nothing to help choose.

Considering the French Revolution question, I can imagine that students may progress on the following ability ladder between the two extremes.

  1. I know it happened some time in the past
  2. There was this bare chested painting of woman on barricade, and lots of beheadings later. No clue when, though.
  3. I know it had something to do with the US Independence war, but I don't know which was first
  4. US Independence war was first
  5. I faintly remember a collection of dates which I memorized for the test, but I don't know which is which.

This will be different for each student, though, so different options are differently likely for each.

In education, there is item response theory, which tries to estimate a model of student's ability by presenting large set of students with large set of questions, and mutually calibrating the questions on students and students on questions. Various models of various complexity can be used to quantify student abilities.

I believe that the question cannot be answered without actual experimentation of some kind (or actual teachers experience substituting for it). If I had the data, I don't know for certain what I'd do with it, but I believe something can be done then.

user7610
  • 206
  • Unfortunately I don't have data about the players' abilities and cannot experiment with them as I generate the questions automatically beforehand, but in that case it seems that there is no general solution. However I would like to view the case more abstractly, without players, as a statistical problem. – Konrad Höffner Apr 15 '19 at 10:09
  • I think you cannot do this without experimentation. You may be able to use experiments done with one set of questions to another set of questions, or substitute common sense and experience, but there has to be some reality check of this sort. If only to evaluate the properties of the choice generation method, to choose the best one. – user7610 Apr 15 '19 at 12:45
  • "I don't have data about the players' abilities and cannot experiment with them as I generate the questions automatically beforehand". The strength of IRT is that you do not need to start with data about players abilities nor about question difficulty. You can estimate them both in tandem from studying how players interact with the questions. – user7610 Apr 15 '19 at 19:21