2

I am a software engineer with a stats problem: I want to deploy a new version of a web application to a subset of users, measure the error rates, and if there is not an increase of error rates, direct it to all users. (We call this 'Canary Releasing' - including this term so the problem is searchable)

I'm having trouble determining if the error rate has increased. My naive approach was to just calculate the error rate of each version:

error rate A = # of errors in A / # requests in A

error rate B = # of errors in B / # requests in B

However this is a bit of a coin flip: the rates are never the same and it's possible that error rate B is higher but it is not significant.

Is there a better test I can use?

Dan Alvizu
  • 121
  • Well, the issue is judging whether the difference in rates is significant, right? – hardmath Jun 16 '16 at 16:16
  • 2
    You should use the raw data before doing the division (a particular difference in rates will be more significant when there are more requests). A chi-squared test will probably meet your needs, or if the samples are very small then a Fisher exact test. – Henry Jun 16 '16 at 16:16

  • Well, the issue is judging whether the difference in rates is significant, right?

    Yes - although I only care if it's different one way (new application has more errors)

     – Dan Alvizu Jun 16 '16 at 16:24
  • I'm not sure you really mean what you're saying, unless your sample sizes are in the thousands. [It takes a public opinion poll of 2500 subjects for the margin of sampling error to be as low as $\pm .02$ (2 percent).] Please provide typical sample sizes for A and B and realistic error rates, with an example where you strongly suspect (intuitively) that new version B has a higher enough error rate to question its use. The topic to google is 'test for difference in proportions'. – BruceET Jun 16 '16 at 21:22

  • Please provide typical sample sizes for A and B and realistic error rate My samples would be 'time slices' of 15 seconds - i'll get a count of total requests and total number of errors in that time. This would go on for about 20 minutes, so 80 data points total. I'll be routing 90% of all requests to version A and 10% to version B. I'm also using this same technique for a number of widely different services with different traffic profiles: some will get thousands of requests, and say 5% of those would generate errors. Others will get dozens of requests, and almost no errors at all.

     – Dan Alvizu Jun 16 '16 at 21:59
  • What you want is called a "test of hypothesis for proportions". Google this phrase and you will get many hits. – awkward Jun 17 '16 at 00:07

1 Answers1

0

A few years later, Netflix open sourced something in this area and decided to use the The Mann-Whitney U test:

https://github.com/spinnaker/kayenta/blob/master/kayenta-judge/src/main/scala/com/netflix/kayenta/judge/classifiers/metric/MannWhitneyClassifier.scala

https://en.wikipedia.org/wiki/Mann%E2%80%93Whitney_U_test

Dan Alvizu
  • 121
  • I realize you are posting a self-answer, but what you have posted little more than links. The links might point to some material that answers your Question, but you should quote the most relevant portion of what can be found there, or at least summarize in your own words what Readers can conclude from that. – hardmath Jan 17 '22 at 04:47