Recently, I posted this question (When is the Law of Large Numbers not True?) where I was interested in creating an example where the Law of Large Numbers (LLN) DOES NOT apply.
Part 1: In short, the idea I came up with was to create a correlated random variable such that the IID/finite variance assumption required for LLN is not met:
Suppose there are $N$ random variables with each pairwise covariance $Cov(X_i, X_j) = c_{ij}$. Note that there will be $\frac{N(N-1)}{2}$ covariance terms.
We can redefine $M = \sum_{{i=1}}^{N} \frac{X_i}{N}$ and the variance of M as:
$$Var(M) = Var\left(\frac{X_1 + X_2 + ... + X_N}{N}\right) = \frac{1}{N^2} \left[ Var(X_1) + Var(X_2) + ... + Var(X_N) + 2Cov(X_1, X_2) + Cov(X_1, X_3) + 2Cov(X_2, X_3) + .... \right] = \frac{1}{N^2} \left( N\sigma^2 + \sum_{{i=1}}^{n} \sum_{{j \neq i, j=1}}^{n} Cov(X_i, X_j) \right)$$
We can apply this result within Chebyshev's Inequality:
$$\lim_{{n \to \infty}} P\left( \left| \overline{X_n} - \mu \right| \geq \epsilon \right) \leq \frac{\frac{1}{N^2} \left( N\sigma^2 + \sum_{{i=1}}^{N} \sum_{{j \neq i, j=1}}^{N} Cov(X_i, X_j) \right)}{\epsilon^2} $$
And from here, it is uncertain whether:
$$\lim_{{N \to \infty}} \sum_{{i=1}}^{N} \sum_{{j \neq i, j=1}}^{N} \frac{Cov(X_i, X_j)}{N^2} = 0 $$
Therefore, in the above case, the Weak Law of Large Numbers does not hold.
My Question: In the previous question (i.e. When is the Law of Large Numbers not True?), I was told that my "example is too broad to say one way or the other. For different choices of distributions (and thus different $c_{ij}$'s, we may have CLT or not have CLT (similarly WLLN)."
Part 2: To make my example more concrete, I thought that perhaps I could see how this example behaves when the correlation increases. For a given correlation (rho) value, random samples are generated from a correlated Normal Distribution (with mean = 0) and the sample means is calculated for each random sample (sample size for each sample changes). This is then repeated for different values of rho.
Here is a quick recap of the Multivariate Normal Distribution:
$$f(\mathbf{x} | \boldsymbol{\mu}, \boldsymbol{\Sigma}) = \frac{1}{\sqrt{(2\pi)^k |\boldsymbol{\Sigma}|}} \exp\left(-\frac{1}{2} (\mathbf{x} - \boldsymbol{\mu})^T \boldsymbol{\Sigma}^{-1} (\mathbf{x} - \boldsymbol{\mu})\right)$$
where:
$\mathbf{x} = [x_1, x_2, ..., x_k]^T$ is a $k$-dimensional real vector.
$\boldsymbol{\mu} = [\mu_1, \mu_2, ..., \mu_k]^T$ is the mean vector where $\boldsymbol{\mu} \in \mathbb{R}^k$.
$\boldsymbol{\Sigma}$ is the covariance matrix where $\boldsymbol{\Sigma} \in \mathbb{R}^{k \times k}$.
The covariance matrix $\boldsymbol{\Sigma}$ is defined as:
$$\boldsymbol{\Sigma} = \begin{bmatrix} \sigma_{1}^2 & \sigma_{12} & \cdots & \sigma_{1k} \\ \sigma_{21} & \sigma_{2}^2 & \cdots & \sigma_{2k} \\ \vdots & \vdots & \ddots & \vdots \\ \sigma_{k1} & \sigma_{k2} & \cdots & \sigma_{k}^2 \end{bmatrix}$$
where $\sigma_{ii}$ is the variance of the $i$-th dimension and $\sigma_{ij}$ (or $\sigma_{ji}$) is the covariance between the $i$-th and $j$-th dimensions.
The correlation between the $i$-th and $j$-th dimensions is given by:
$$\rho_{ij} = \frac{\sigma_{ij}}{\sqrt{\sigma_{ii} \sigma_{jj}}}$$
Part 3: I made the following simulation using the R programming language for a 2 dimensional normal distribution:
set.seed(123)
mu <- 0
sigma <- 1
N <- 500
correlations <- seq(0, 1, by = 0.1)
sample_means_df <- data.frame()
for (rho in correlations) {
# Generate two sets of random numbers with the specified correlation
Sigma <- matrix(c(sigma^2, rhosigma^2, rhosigma^2, sigma^2), nrow=2)
data <- MASS::mvrnorm(N, mu = c(mu, mu), Sigma = Sigma)
sample_means <- numeric(N)
for (n in 1:N) {
sample_means[n] <- mean(data[1:n,])
}
sample_means_df <- rbind(sample_means_df, data.frame(SampleSize = 1:N, SampleMean = sample_means, Correlation = rho))
}
library(ggplot2)
ggplot(sample_means_df, aes(x = SampleSize, y = SampleMean, color = Correlation)) +
geom_line() +
geom_hline(aes(yintercept = mu), color = "black") +
scale_color_gradientn(colors = rainbow(5), name = "Correlation") +
labs(x = "Sample Size", y = "Sample Mean",
title = "Sample Mean vs. Sample Size for Different Correlations",
subtitle = paste("Population Mean:", mu)) + theme_bw()
In the above plot, empirically, we can see that for larger correlations values (i.e. purple, blue colors), the sample mean takes a much longer time to approach the true mean compared to smaller correlations values (i.e. red, yellow).
Mathematically, this suggests that the limit in the presence of larger correlation values takes longer to approach 0. Thus:
- Is my understanding of this correct?
- Have I fully defined an example where LLN does not apply?
Thanks!
Note: A comparison of the 0 correlation case vs the 0.95 correlation case (0 correlation approaches true value faster)
set.seed(123)
mu <- 0
sigma <- 1
N <- 1000
correlations <- c(0, 0.95)
correlation
sample_means_df <- data.frame()
for (rho in correlations) {
correlation
Sigma <- matrix(c(sigma^2, rhosigma^2, rhosigma^2, sigma^2), nrow=2)
data <- MASS::mvrnorm(N, mu = c(mu, mu), Sigma = Sigma)
sample_means <- numeric(N)
for (n in 1:N) {
sample_means[n] <- mean(data[1:n,])
}
sample_means_df <- rbind(sample_means_df, data.frame(SampleSize = 1:N, SampleMean = sample_means, Correlation = rho))
}
library(ggplot2)
ggplot(sample_means_df, aes(x = SampleSize, y = SampleMean, color = as.factor(Correlation))) +
geom_line() +
geom_hline(aes(yintercept = mu), color = "black") +
scale_color_manual(values = c("red", "blue"), name = "Correlation") +
labs(x = "Sample Size", y = "Sample Mean",
title = "Sample Mean vs. Sample Size for Different Correlations",
subtitle = paste("Population Mean:", mu)) + theme_bw()
