2

I have three techniques, called A, B and C. Each can be used independently when trying to perform four related tasks (Tasks 1, 2, 3 and 4). I have run lots of tests, and tried all combinations of each technique being on or off. My results look something like this. Let's say that higher numbers are better.

$$ \begin{array}{l|r|r|r|r|r|r|r|r|r|} \mbox{Technique $A$} & - & - & - & - & X & X & X & X \\ \mbox{Technique $B$} & - & - & X & X & - & - & X & X \\ \mbox{Technique $C$} & - & X & - & X & - & X & - & X \\ \hline \mbox{Task $1$} & 433 & 277 & 911 & 492 & 686 & 4211 & 3775 & {\bf 9732}\\ \mbox{Task $2$} & 149 & 1063 & 5562 & {\bf 6035} & 3 & 58 & 1391 & 1708\\ \mbox{Task $3$} & 220 & 1278 & 7014 & {\bf 7018} & 10 & 97 & 2083 & 4452\\ \mbox{Task $4$} & 218 & 1255 & 6142 & {\bf 8656} & 1 & 73 & 1087 & 2056\\ \end{array} $$

Looking at the numbers, it seems that $B+C$ is good for Tasks 2,3 and 4, and that $A$ on its own is best for Task 1. But I want to say a bit more. I'd like to be quantitative if I can. My question is: can I deduce anything quantitative from this data? Or do I really need some measure of the variance of the observations? That is, I suspect the numbers might be different if I ran all the tests again.

John Wickerson
  • 431
  • This question is better suited to the stats site – Sergio Parreiras Jul 28 '14 at 13:50
  • @SergioParreiras Thanks, I didn't know about that site. Might a mod be kind enough to migrate the question for me? – John Wickerson Jul 28 '14 at 13:51
  • 1
    My suggestion is to copy and past there, it is faster. – Sergio Parreiras Jul 28 '14 at 13:54
  • What do the numbers in the table exactly represent? In particular, can we interpret them as "percentages of success", or as scores obtained in other ways? – Anatoly Jul 29 '14 at 16:34
  • @Anatoly They mean "number of successes out of 100000 attempts". So they're percentages if you divided them by 1000 :-). By the way, in the interests of full disclosure, I should probably mention that I followed Sergio's advice and have cross-posted on Cross-Validated. (http://stats.stackexchange.com/questions/109803/what-do-i-need-in-order-to-draw-conclusions-from-this-data) – John Wickerson Jul 29 '14 at 16:36

1 Answers1

1

Since you are exploring a categorical variable (success-failure), the appropriate test is a chi-square test. You have to perform separate analyses for the four tasks. For each task, build a $2\times8$ contingency table where the 8 columns represent the different method (single or combined techniques) and the 2 rows represent successes and failures. Then fill each cell with the number (not percentage) of successes and failures, respectively (the latter are clearly given by 100000 - number of successes), for each method. Lastly, run the chi-square test for this $2\times8$ table using one of the several statistical softwares or online tools. Repeat this analysis for each task.

For tasks where you get a significant p value, you can then use some additional procedures for pair-wise comparisons of proportions. These methods, (e.g., the Marascuilo procedure, the "partitioned chi square" approach, the Holm–Bonferroni method, and so on) allow multiple comparisons for proportions when we want to test which particular proportions are different from each, and can be used when the null hypothesis has been rejected an overall chi-square test. This further analysis could give you additional information on what is the best method.

Anatoly
  • 17,079