2

We are measuring the number of times a certain event happens. We do this with the help of sampling, so that we only report events with a probability p. For example p=0.01 would result in about 1/100 events coming through. Now, if we have measured N events, we will assume that there have for real been N/p events.

We would be interested to know what the error margin is based on p and N. We would like to be abled to say that with 95% probability our result is within some margin of error. How to calculate this? We are looking for a function which takes N and p as parameters.

Thanks for your help in advance!

Frans Ekman
  • 21

1 Answers1

2

I think this question fits more into Crossvalidated or Statistics part of SE, but here is how I would deal with this issue:

Number of events, or count data, is commonly modeled with Poisson distribution, which takes just one parameter - $\lambda$ (lambda). Poisson distribution's mean and variance equals to this $\lambda$.

So, for example, if you measured this N for several periods, and have, e.g. a vector of Ns [1000, 1020, 1040, 970, 900]; In R, to get $\lambda$ given such a vector, you would do:

library(MASS)
Ns <- c(1000, 1020, 1040, 970, 900)
params <- fitdistr(Ns, "poisson") # fit poisson distribution
lambda <- params$estimate # get mean 
sd <- params$sd # get standard deviation

Then, if your sample is large, you can get 95% confidence interval's lower and higher boundaries by multiplying standard deviation with 1.96 and subtracting and adding it from mean respectively. 

ci <- c(lambda + c(-1,1) * 1.96 * sd)

Edited: forgot to mention, now that you have 95% CI, you can answer your question by checking if the value you are interested in is within this 95% CI or not.

As for the sampling rate, I'm not sure if it really matters, if you know for sure that you sample every 100th value - just multiply the values in Ns by 100.

Edited: Maybe this helps also: https://en.wikipedia.org/wiki/Binomial_proportion_confidence_interval

p <- 0.01 # expected proportion of logged events
N <- 600 # number of logged events
Np = N/p # expected number of events (sample size)
# For each event we have a Bernoulli trial, if it gives 1 - we record, if 0 - we ignore.
# Then N is the total heads.
# We can get 95%  confidence intervals for proportion of logged events p:  
ci <- c(p + c(-1,1) * 1.96 * sqrt((1/Np)*p*(1-p)))

Or given your N and p, what would be the uncertainty about the p.

Dmitry Smirnov
  • 121
  • Thanks for your answer. The problem is that in many cases we end up getting very few events and the world changes over time. So we cannot collect events over several periods and calculate standard deviation. However, we could have events sent to several buckets and calculate standard deviation from that, but then we would be sending a lot more data (as many times more as we have buckets). Therefore we wonder if there is a formula for this. – Frans Ekman Sep 07 '15 at 14:50
  • Regarding "if you know for sure that you sample every 100th value - just multiply the values in Ns by 100", the sampling must be probabilistic, since the logic sending the events cannot be aware of when other events have been sent. – Frans Ekman Sep 07 '15 at 14:51
  • If I understand you correctly, you have some process that generates data, and with probability p you take out a sample from this process? Do you know the total amount of events, from which the random draws were taken? Let's say, this process was going on for a day, what would be the N? What would you expect it to be? I guess using Poisson process will provide more info: http://www.stat.nus.edu.sg/~staxyc/spchapt3a.pdf I'm not very experienced in that, but if you answer my above questions, I can try to look into it more :) – Dmitry Smirnov Sep 07 '15 at 20:32
  • No, we do not know the total number. The total number of events is what we try to find out together with its error margin. – Frans Ekman Sep 08 '15 at 06:52
  • I think this may be an ill-posed problem. Am I understanding it right that you don't know the real p, and have a guess of possible p? This way, given N, there is infinite number of combinations of p and total N that could have produced that N. If you go with Poisson, you can count the events and get underlying distribution parameters, but you can't infer sample size. If you sample e.g. every day, you can use Poisson model, get lambda estimate and its variation, and check if new samples provide evidence for change in lambda, and this gives you some information about underlying process. – Dmitry Smirnov Sep 08 '15 at 13:41
  • No, we do know p. We also know how many events we received (obviously). But we do not know how many there were for real. The formula you suggested may be the right one and probably is. I would have to do a sanity check and artificially produce some tests where I know the total number of events and check what N I get and how often it falls within the error margin produced by your formula. Thanks for your help! – Frans Ekman Sep 09 '15 at 14:58