I am trying to create a mathematical example where the Weak Law of Large Numbers (WLLN) does not apply.
Part 1: Here is the basic definition for the WLLN. Suppose we have $n$ observations from a random variable $X$ . Let's define $\overline{X_n}$ = $\frac{1}{n} \sum_{{i=1}}^{n} x_i$ and $\mu$ as the true value of $\overline{X_n}$. Let's also assume that $X$ has finite variance:
Then the WLLN states that:
$$\lim_{{n \to \infty}} P\left( \left| \overline{X_n} - \mu \right| \geq \epsilon \right) = 0$$
Part 2: Using Chebyshev's Inequality (https://en.wikipedia.org/wiki/Chebyshev%27s_inequality), we can prove the WLLN.
Chebyshev's Inequality is for an individual point $x$. That is, Chebyshev's Inequality describes the probability of an individual point from a random variable subtracted by the mean of the random variable (assuming that the random variable has some fixed variance $\sigma$):
$$\text{Chebyshev's Inequality }: \quad \lim_{{n \to \infty}} P\left( \left| x - \mu \right| \geq k\sigma\right) \leq \frac{1}{k^2}$$
If we re-define $k \sigma = \epsilon$ and $k = \frac{k}{\sigma}$, then we can re-write Chebyshev's Inequality as:
$$\lim_{{n \to \infty}} P\left( \left| x - \mu \right| \geq \epsilon\right) \leq \frac{\sigma^2}{\epsilon}$$
From first principles, we can define the variance of $\overline{X_n}$:
$$Var(\overline{X_n}) = \frac{1}{N^2} \sum_{{i=1}}^{N} x_i = \frac{N \sigma^2}{N^2} = \frac{\sigma^2}{N}$$
We can now use this observation and update Chebyshev's Inequality:
$$\lim_{{n \to \infty}} P\left( \left| \overline{X_n} - \mu \right| \geq \epsilon \right) \leq \frac{Var(\overline{X_n})}{\epsilon^2} = \frac{\sigma^2}{N\epsilon^2} $$
Evaluating the limit, we see that:
$$\lim_{{n \to \infty}} \frac{\sigma^2}{N\epsilon^2} = 0$$
This means that as the number of observations used to calculate $\overline{X_n}$ becomes larger and larger, the probability of $\overline{X_n} = \mu$ also becomes larger and larger. This concludes the proof of the WLLN.
Part 3: Here is my logic: If I want to create an example where WLLN does not hold, then this case should "violate" Chebyshev's Inequality.
The example I thought of involves $X$ having non-zero covariances. Specifically:
$$Var(\overline{X_n}) = Var(\frac{1}{N} \sum_{{i=1}}^{N} x_i) = \frac{1}{N^2} \sum_{{i=1}}^{N} \sum_{{j=1}}^{N} Cov(X_i, X_j) $$
$$ Cov(X_i, X_i) = Var(X_i) = \sigma^2 $$
Thus, suppose I define two random variables $X_1$ and $X_2$ such that $Cov(X_1, X_2) = c$. If I define $M = \frac{X_1 + X_2}{2}$, then I can evaluate the Variance of $M$ as (assume the true mean of $M$ is $\mu$):
$$Var(M) = Var\left(\frac{X_1 + X_2}{2}\right) = \frac{1}{4} \left[ Var(X_1) + Var(X_2) + 2Cov(X_1, X_2) \right] = \frac{1}{4} \left( 2\sigma^2 + 2c \right) = \frac{\sigma^2}{2} + \frac{c}{2} = \frac{\sigma^2 + c}{2}$$
Now, suppose there are $N$ random variables with each pairwise covariance $Cov(X_i, X_j) = c_{ij}$. Note that there will be $\frac{N(N-1)}{2}$ covariance terms.
We can redefine $M = \sum_{{i=1}}^{N} \frac{X_i}{N}$ and the variance of M as:
$$Var(M) = Var\left(\frac{X_1 + X_2 + ... + X_N}{N}\right) = \frac{1}{N^2} \left[ Var(X_1) + Var(X_2) + ... + Var(X_N) + 2Cov(X_1, X_2) + Cov(X_1, X_3) + 2Cov(X_2, X_3) + .... \right] = \frac{1}{N^2} \left( N\sigma^2 + \sum_{{i=1}}^{n} \sum_{{j \neq i, j=1}}^{n} Cov(X_i, X_j) \right)$$
We can apply this result within Chebyshev's Inequality:
$$\lim_{{n \to \infty}} P\left( \left| \overline{X_n} - \mu \right| \geq \epsilon \right) \leq \frac{\frac{1}{N^2} \left( N\sigma^2 + \sum_{{i=1}}^{N} \sum_{{j \neq i, j=1}}^{N} Cov(X_i, X_j) \right)}{\epsilon^2} $$
And from here, it is uncertain whether:
$$\lim_{{N \to \infty}} \sum_{{i=1}}^{N} \sum_{{j \neq i, j=1}}^{N} \frac{Cov(X_i, X_j)}{N^2} = 0 $$
Therefore, in the above case, the Weak Law of Large Numbers does not hold.
My Question: Can someone please tell me if my analysis is correct? Have I correctly created an example where the Weak Law of Large Numbers does not hold?
Thanks!
