0

I'm planning to write a small app for a tedious task:

  • Generating pdf based on a existing template (.txt basically)
  • Binding specific data (some text, recipient, etc) to that template

Suppose I have 100 pdf to generate. I expect $±100$% confidence if I check manually all the generated pdf.

  • How many pdf should I check to reach a level of confidence of $95$%, $99$%, $99,999$%?
  • What additional parameter should I deal with?

At first sight, checking only a few could give me a high confidence of reliability. However I'm wondering how to rigorously achieved such confidence using a proper formula.

Thanks for your help.

Irregardless
  • 469
roland
  • 137

1 Answers1

1

Here you can find several ways of calculating confidence intervals, wich I'm talking about. https://en.wikipedia.org/wiki/Binomial_proportion_confidence_interval

I calculate confidence intervals for probability that your program works properly. Here a my results for different types of one-sided 95% confidence intervals, if you check 100 out of 100.

Jeffrey (0.97525, 0.99999)
Agresti-Coull (0.9555, 1.0074)
Direct stepped binomial test (0.96244, 1)
Wilson (0.96300, 1.0)
Beta (0.96378, 1.0)

Edit

These estimates are for the probability that your program works correctly for the random document from the infinite set of possible documents, considering 100 of 100 has been processed correctly.

I've calculated these intervals with python library statmodels using fuction proportion_confint.

If you want confidence intervals for the probability, that your program will work for 100 of 100, having that it has worked for $x$ of $x$, that would be more difficult to calculate. The results for different documents may be highly correlated. Though if we consider them independent, we can go by the following algorithm:

  1. choose the confidence level $c$.
  2. calculate $c' = c^{1/(100-x)}$
  3. calculate the lower bound $lb$ for probability of success of one calculation with the algorithms mentioned above on the level $c'$.
  4. the resulting interval is $(lb^{100-x}, 1)$

For example for $x = 80$ and $c=0.95$, the resulting interval would only be $(0.1846, 1)$. With increasing $c$ and decresing $x$, the $lb$ will only get smaller. Because this is the confidence interval for the probability that all $(100-x)$ of your calculations would be successful.

Dimitrius
  • 283
  • 1
  • 6
  • I don't understand. If I check 100 out of 100, my confidence is 100%. How can I calculate each intervals in the results you mentioned? – roland Jul 30 '17 at 20:08
  • @roland For what value do you want to estimate the confidence intervals? I propose estimates for the probability that your programm works correctly for the random document from the infinite set of possible documents, considering 100 of 100 have been processed correctly. – Dimitrius Jul 31 '17 at 12:03
  • I cannot get 0.78. What I did: (0.8 exp (1/20)) exp 20 = 0.80 – roland Jul 31 '17 at 16:01
  • 1
    @roland You're right, I miscalculated. I calculated confidence level $c' = 0.95^{1/20} = 0.99744$. Than for exact stepped interval (you can find it here. Confidence level is at the bottom, don't miss it http://statpages.info/confint.html) $lb = 0.91899$ and than interval lower bound is $lb^{20} = 0.1846$. But as I said the interval gives estimate for independent results, and in reality they are highly dependent and result would be much higher. – Dimitrius Jul 31 '17 at 16:30