Average Distance of the Sample Mean from the True Mean?

Question

This is a question I have been thinking of:

Suppose I have a Normal Distribution with a specific mean (e.g. "a") and standard deviation (e.g. "b") - if I draw "n" random numbers from this distribution and take the mean of these "n" numbers : on average, how close will this mean be from "a"?

For example, using the R programming language, I tried to run this simulation:

set.seed(123)
results = list()
for (i in 1:1000)
{
n = 100, a = 5, b = 5
sample_i =  rnorm(100, 5, 5)
mean_i = mean(sample_i)
difference_i = abs(5 - mean_i)
results[[i]] = data.frame(i,difference_i)
}
final = do.call(rbind.data.frame, results)
plot(density(final$difference_i), main = "Spread of Errors : n = 100, a = 5, b = 5")

I can now show this for n = 1000:

results = list()
for (i in 1:1000)
{
n = 1000, a = 5, b = 5
sample_i =  rnorm(100, 5, 5)
mean_i = mean(sample_i)
difference_i = abs(5 - mean_i)
results[[i]] = data.frame(i,difference_i)
}
final = do.call(rbind.data.frame, results)
plot(density(final$difference_i), main = "Spread of Errors : n = 1000, a = 5, b = 5")

My Question: In general, given a specific probability distribution - is there some mathematical formula which shows on average, how far the mean from a sample of size "n" will deviate from the true mean of this specific probability distribution?

Thanks!

EDIT - NOTE:

As a concrete example :

• Consider 1000 random draws from a Normal Distribution with Mean=a and Standard_Deviation = b : On average, what will be the expected difference between the mean of these 1000 random draws and the true mean (i.e. "a")?

• Consider 1000 random draws from a Poisson Distribution with the Rate_Parameter = "lambda: : On average, what will be the expected difference between the Rate Parameter calculated from these 1000 random draws and the true Rate Parameter (i.e. "lambda")?

• In general, for "n" random draws from some general probability distribution - how will the mean calculated from these "n" random draws differ from the true mean of this distribution (on average)? Is there a mathematical formula that can be used to describe this relationship? (e.g. via Central Limit Theorem)

Answer 1

Given the random variable $X$ with expectation $\mu$ and standard deviation $\sigma$, the average deviation of the sample mean $\bar X = \frac{1}{N} \sum_{i=1}^N X_i $ from the "true" population mean $E[\bar X] = E[X] = \mu$ is by definition the standard deviation of the sample mean as follows

$$\sigma_{\bar X} = \sqrt{Var[\bar X]} = \sqrt{E[(\bar X - \mu)^2]} = \frac{\sigma}{\sqrt N}$$

This is because the standard deviation of the sample mean is the square root of a weighted average of the squared deviations between the sample mean and the true mean. Of course, each deviation is weighted by the probability of its corresponding sample mean. The above equation is true regardless of the function form of the probability distribution of $X$. This is a direct result of the central limit theorem which implies the sample mean $\bar X$ is always normally distributed and increasingly so for large $N$.

Answer to Edit

As a concrete example :

• Consider 1000 random draws from a Normal Distribution with Mean=a and Standard_Deviation = b : On average, what will be the expected difference between the mean of these 1000 random draws and the true mean (i.e. "a")?

$$ \frac{b}{\sqrt{1000}} = \frac{b}{10 \sqrt{10}}$$

• Consider 1000 random draws from a Poisson Distribution with the Rate_Parameter = "lambda: : On average, what will be the expected difference between the Rate Parameter calculated from these 1000 random draws and the true Rate Parameter (i.e. "lambda")?

Note that the rate parameter lambda $\lambda$ is the expectation of the Poisson distribution and hence the "true" population mean, so this question is no different than the first one above. However, in order to solve it you must recognize that the Poisson distribution has the unique property where the mean and variance are equal. In other words, $ E[X] = a = Var[X] = b^2 = \lambda $. In turn, this implies the standard deviation of the Poisson distribution is $b = \sqrt \lambda$. You now have what you need to solve the problem. The Central Limit Theorem tells us the sample mean will be normally distributed even if the underlying probability distribution is Poisson, so the standard deviation of the sample mean is given by

$$ \frac{\sqrt \lambda}{\sqrt{1000}} = \frac{\sqrt \lambda}{10\sqrt{10}} $$

which is no different from above. We are still dividing the standard deviation by the square root of the sample size.

• In general, for "n" random draws from some general probability distribution - how will the mean calculated from these "n" random draws differ from the true mean of this distribution (on average)? Is there a mathematical formula that can be used to describe this relationship? (e.g. via Central Limit Theorem)

This would be the equation i gave you above...

$$ \frac{b}{\sqrt{n}}$$