1

Hypergeometric distribution describes the probability of k successes in n draws from population of N with K successes. But what if I want expected number of draws until k successes are drawn from the same population?

  • Wider question: in a population of N is 1 ill person. At each moment of time, two people meet. If one of them is ill and the other healthy, the healthy gets ill. What is the expected time count until at least n people are ill? – Miloš Lukačka Jan 07 '18 at 11:32
  • So, in you case you need to apply the following trick: you keep drawing until $k$ successes, you last draw (the $k$-th one) is a success, the remaining $k-1$ are scattered across $n-1$ draws. Or $P(n \text{ draws until }k\text{ successes})=P(\text{last draw is success})\cdot P(n-1 \text{ draws have }k-1\text{ successes})$ – rtybase Jan 07 '18 at 11:43
  • 1
    okay but the expected value of the second factor is still dependent on n, specifically it is (n-1)K/N – Miloš Lukačka Jan 07 '18 at 11:59
  • 1
    As per my comment $$P(n \text{ draws until }k\text{ successes})=P(\text{last draw is success})\cdot P(n-1 \text{ draws have }k-1\text{ successes})$$

    $P(n-1 \text{ draws have }k-1\text{ successes})$ is the standard hyper geometric probability $$P(n-1 \text{ draws have }k-1\text{ successes})=\frac{\binom{K}{k-1}\binom{N-K}{n-k+1}}{\binom{N}{n-1}}$$ and (everything is without replacement) $$P(\text{last draw is success})=\frac{1}{K-k+1}$$ And $$E[X]=\sum\limits_{n\geq k}nP(n \text{ draws until }k\text{ successes})$$

    – rtybase Jan 07 '18 at 12:03
  • ah! you can use the infinite sum to get the expected value, thanks! if you submit this as an aswer, I will gladly accept it. – Miloš Lukačka Jan 07 '18 at 12:07
  • respectively, the upper bound of the sum is N, since you can't draw more than the whole population, right? – Miloš Lukačka Jan 07 '18 at 12:09
  • Yes, but the trick is, you technically don't depend on that. You can extract $P(\text{last draw is success})=\frac{1}{K-k+1}$ out of the summation, as it doesn't depend on $n$ and the expectation calculation reduces to calculating expectation of hyper geometric distribution for $n-1$. – rtybase Jan 07 '18 at 12:10
  • but the expected value of the standard n-1 hypergeometric distribution is still dependent on the number of draws, which is my unknown – Miloš Lukačka Jan 07 '18 at 12:22
  • True, well, it seems I was wrong with "calculation reduces to calculating expectation of hyper geometric distribution for $n−1$" assumption. For hyper geometric distribution, expectation is the number of successes out of $n$ (known) draws. Well, that sum will have to be evaluated differently then (don't have an answer now, but we have the expected value formula)

    $$E[X]=\sum\limits_{n\geq k}nP(n \text{ draws until }k\text{ successes})=\sum\limits_{n\geq k}n\frac{\binom{K}{k-1}\binom{N-K}{n-k+1}}{\binom{N}{n-1}}\frac{1}{K-k+1}$$ $E[X]$ is a function of $k$

    – rtybase Jan 07 '18 at 12:31
  • well I could just solve the sum numerically, since we know lower and upper bound. I will compare it to the empirical results I got to see if it's correct – Miloš Lukačka Jan 07 '18 at 12:33
  • Cool, don't forget to drop a message with the results ;) – rtybase Jan 07 '18 at 12:34
  • Another typo in my comments $$P(\text{last draw is success})=\frac{K-k+1}{N-n+1}$$ – rtybase Jan 07 '18 at 13:35
  • it doesn't really matter, those combination numbers are too big to be solved numerically.. – Miloš Lukačka Jan 07 '18 at 13:45
  • in my model, the population is N=10000 and at the beginning is K=1 infected person. At each moment 2 people meet, so if I want number of moments until 2nd person will get infected - that is E[X], k = 1, K = 1. When I want number moments from 2nd infected until 3rd person gets infected, it is E[X], k=1, K=2 etc. The number of moments until 1000 people get infected is sum of E[X] for k=1, K=1...999, but those sums are just too big, there has to be another way – Miloš Lukačka Jan 07 '18 at 13:49
  • my another idea was looking at expected values: expected value of binomial is E=np, so when I want number of steps until first success, it is n=1/p. Taking this to hypergeometric distribution, expected value is E=n*K/N, so it could work as number of steps for 1st success is n=N/K, which makes sense but it doesn't add up compared to my results – Miloš Lukačka Jan 07 '18 at 13:52
  • I know that the expected number of draws in the binomial distribution is because negative binomial distribution, resp. geometric distribution, but still ... :) – Miloš Lukačka Jan 07 '18 at 13:55
  • If the numbers are big, the expectation contains lots of combinations, which it turns contain factorials. You can approximate them using Stirling's approximation – rtybase Jan 07 '18 at 14:02
  • I used Stirling, still the numbers are just too big. I am trying to simplify the whole thing, but I keep getting NaNs – Miloš Lukačka Jan 07 '18 at 14:55
  • That's the "exciting" part ... here is a similar topic, at least we can confirm that the logic is correct. – rtybase Jan 07 '18 at 15:06
  • okay, assuming k=1, I was able to put it down to K((N-n)...(N-n-K+1))/(N)...(N-K+1) but it still gives NaNs for K bigger than ~75 and I need the function for K=1000 ... – Miloš Lukačka Jan 07 '18 at 15:57
  • What programming language do you use? – rtybase Jan 07 '18 at 16:08
  • I am trying to do it in R – Miloš Lukačka Jan 07 '18 at 16:11
  • also, that estimation doesn't work for K>2, cause I can't simplify the factorials the way I did for K=1 – Miloš Lukačka Jan 07 '18 at 16:23
  • I'd say, try Python, it supports really large data types – rtybase Jan 07 '18 at 16:31
  • so the problem is the last fraction with combination numbers in both nominator and denominator. I know that the nominator will be like (N-K) times .... times (N-K-n+1) and the denominator like (N) times ... times (N-K+1). So in both there will be some factors, that can be factored out. Right now i am trying to store all the factors in fields, compare the fields and if there is factor in both, reduce it to 1. But it's painfully slow... – Miloš Lukačka Jan 07 '18 at 16:41
  • I dont think any language will handle (10000 choose 9124).. – Miloš Lukačka Jan 07 '18 at 16:44
  • try https://www.tutorialspoint.com/execute_python_online.php with $$ \text{from scipy.special import comb} \ \text{print comb(10000, 9124, exact=True)} $$ – rtybase Jan 07 '18 at 16:53
  • I don't think there is a point of continuing in this. Even after heavy optimization I still can't get expected value for 1 person to get sick faster than ~10 minutes, doing this for 1000 people and considering the computations will be more difficult with bigger K, I won't have a solution under 10 hours – Miloš Lukačka Jan 07 '18 at 18:52
  • I've tried the estimation for smaller N and it works really fine, I just don't have enough power. Thanks again for your help. – Miloš Lukačka Jan 07 '18 at 19:15

1 Answers1

0

As per my comments $$P(n \text{ draws until }k\text{ successes})=P(\text{last draw is success})\cdot P(n-1 \text{ draws have }k-1\text{ successes})$$

$P(n-1 \text{ draws have }k-1\text{ successes})$ is the standard hyper geometric probability $$P(n-1 \text{ draws have }k-1\text{ successes})=\frac{\binom{K}{k-1}\binom{N-K}{n-1-k+1}}{\binom{N}{n-1}}=\frac{\binom{K}{k-1}\binom{N-K}{n-k}}{\binom{N}{n-1}}$$ and (everything is without replacement) $$P(\text{last draw is success})=\frac{K-k+1}{N-n+1}$$ $E[X]$ is a function of $k$, also considering $\binom{n}{k}=\binom{n}{n-k}$ $$\color{red}{E[X]=}\sum\limits_{n\geq k}nP(n \text{ draws until }k\text{ successes})=\sum\limits_{n\geq k}n\frac{\binom{K}{k-1}\binom{N-K}{n-k}}{\binom{N}{n-1}}\frac{K-k+1}{N-n+1}=\\ \sum\limits_{n\geq k}n\frac{\binom{K}{K-k+1}\binom{N-K}{n-k}}{\binom{N}{N-n+1}}\frac{K-k+1}{N-n+1}=\\ \binom{K}{K-k+1}(K-k+1)\sum\limits_{n\geq k}n\frac{\binom{N-K}{n-k}}{\binom{N}{N-n+1}(N-n+1)}=\\ \frac{K!}{(K-k+1)!(k-1)!}(K-k+1)\sum\limits_{n\geq k}n\frac{\binom{N-K}{n-k}}{\frac{N!}{(N-n+1)!(n-1)!}(N-n+1)}=\\ \frac{K!}{(K-k)!(k-1)!}\sum\limits_{n\geq k}n\frac{\binom{N-K}{n-k}}{\frac{N!}{(N-n)!(n-1)!}}=\\ k\frac{K!}{(K-k)!k!}\sum\limits_{n\geq k}\frac{\binom{N-K}{n-k}}{\frac{N!}{(N-n)!n!}}= \color{red}{k\binom{K}{k}\sum\limits_{n\geq k}\frac{\binom{N-K}{n-k}}{\binom{N}{n}}}$$

I have tried the following Python $3$ code

from scipy.special import comb
from decimal import *

N = 10000
K = 1000

k = 1000

res = Decimal(k * comb(K, k, exact=True))
sum = Decimal('0.0')
one = Decimal('1.0')

def combinations(b, a):
    if (b >= a):
        return comb(b, a, exact=True)
    else:
        return 0

for n in range(k, N - K + k + 1):
    sum = sum + (Decimal(combinations(N-K, n-k)) * one) / Decimal(combinations(N, n))

res = res * sum
print(res)

producing

k=1     E[X]=9.991008991008991008991008979
k=10    E[X]=99.91008991008991008991008991
k=100   E[X]=999.1008991008991008991008979
k=1000  E[X]=9991.008991008991008991008992

all within 8 minutes (altogether, so $\approx 2$ minutes per run) on my MacBook Pro. And

N = 10000 K=5000 k=3000 E[X]=5999.400119976004799040191959

took less than $2$ minutes. Larger values of $N$ take longer to compute (E.g. $N=100000$ I had to stop the program after 1 hour).

Although, it looks like empirically $$E[X]\approx k\cdot \frac{N}{K}$$

rtybase
  • 16,907
  • yes, I had a very similar code, although my CPU isn't as powerful. Considering I would have to evaluate E[X] for all K in 1...1000, it would take hours, but the principle works. – Miloš Lukačka Jan 09 '18 at 20:02
  • I think, it is worth focusing on proving the empirical result. – rtybase Jan 09 '18 at 20:11