Kolmogorov–Smirnov 2-sample test with large sample sizes always significant

Question

I'm aware that the probability of a traditional statistical test such as student's t or mann-whitney u being deemed significant approaches 1.0 as sample size increases (i.e. >10,000) but I'm getting the same issue with a Kolmogorov–Smirnov 2-sample test which I'm having trouble understanding. Doesn't it always evaluate the difference between two sets of 100 cumulative probability values? I don't understand how sample size affects the result.

In theory, the K-S test uses all of the observations available. In practice, many statistical software programs limit the sample size to several thousand to avoid storage and computational difficulties. // The power increases with sample size, so it would not be surprising to reject the null hypothesis (populations the same) unless data in the two groups come from exactly the same population. — BruceET, Nov 27 '17 at 23:26

BruceET · Answer 1 · 2017-11-28T00:09:36.910

Comment continued:

I generated two samples ($X$ and $Y$) of $n = 5000$ each from $\mathsf{Norm}(100, 15)$ and a third ($Z$) of the same size from $\mathsf{Norm}(100, 14).$

Results from the K-S test in R statistical software (where 5000 is the size limit) are as shown below. (With a different seed you would get somewhat different samples, hence somewhat different results.)

set.seed(1017)
x = rnorm(5000,100,15);  y = rnorm(5000,100,15);  z = rnorm(5000,100,14)
ks.test(x, y);  ks.test(x, z);  ks.test(y, z)


        Two-sample Kolmogorov-Smirnov test
data:  x and y 
D = 0.0126, p-value = 0.8222           # Same population, not rejected
alternative hypothesis: two-sided 


        Two-sample Kolmogorov-Smirnov test

data:  x and z 
D = 0.0242, p-value = 0.107            # slightly different pop's, nearly rej
alternative hypothesis: two-sided 


        Two-sample Kolmogorov-Smirnov test

data:  y and z 
D = 0.0312, p-value = 0.01539          # slightly different pop's, rejected
alternative hypothesis: two-sided

Boxplots of the examples are very similar, except perhaps for fewer, less-extreme outliers in the third.:

Below the ECDFs of each sample are compared with the CDF of $\mathsf{Norm}(100, 15)$ (heavy, dashed curves). In the third plot maybe you can see a slight discrepancy.

Kolmogorov–Smirnov 2-sample test with large sample sizes always significant

1 Answers1