1

I recently inherited some code that does bounds checking from a coworker no longer with the company. It has the following comment

//Currently, we use 3 sigma(standard deviation) as upper bound, where
// sigma = SQRT(mean * (1 - mean) / (sample size)).

And the following implementation is supposed to represent the upper bound to check against

MEAN + 3 * SQRT(MEAN * (1 - MEAN) / COUNT)

Does either the formula or the implementation seem correct? I'm not that familiar with Standard Deviation, but the Wiki article on the topic mentions a much different formula.

Kobaj
  • 111

1 Answers1

1

I'll just explain the topic in a little bit more depth than needed but just so you understand fully.

First off for independent random variables, $X_1,...,X_n$ that are from same distribution (you can think of this as sample of data) where $\mu_X=E(X_i)$ and $Var(X_i)=\sigma_X^2$. Now, for the arithmetic average/mean $\overline{X}=\frac{1}{n}\sum_{i=1}^{n}X_i$. Now, one important theorem you should know is the Centeral Limit Theorem (CLT) (this says that sum of iid random variables has an approximate normal distribution for large $n$ ie sample size). Thus CLT asserts that $\overline{X}$ has a normal distribution where $\mu=\mu_X$ and $\sigma^2=\frac{\sigma_X^2}{n}$.

Now, I think the notion of upper bound that you are using is a little misleading. The bound you are using is a the upper bound of a 99% confidence interval (which should not be interpreted as an upper bound for $\overline{X}$. This bound is $$\mu+3\sigma=\mu_X+3\frac{\sigma_X}{\sqrt{n}}$$

Now this is were your problem lies. $\frac{\sigma_X}{\sqrt{n}}\approx \sqrt{\frac{(1-\overline{X})\overline{X}}{n}}$ is only true when $\overline{X}$ represents a proportion (this standard deviation for a Bernoulli random variable

Kamster
  • 1,970
  • 16
  • 18