Degree of freedom in statistic

Question

I am always having difficulty understanding what degree of freedom really is. I know that when I post this question people are going to tell me to do some research myself but I did it and I am still having problem understanding it. I watched youtube videos and read articles ...etc but I still do not understand.

To be specific, let me tell you where I am stuck.

For example

If we have a sample size of 3

{x1,x2,x3} is the set of sample values

At the same time, we assume that we know the sample mean is going to be equal to be 2

knowing there are 3 variables, the first 2 variables can be anything.

Say,

x1 = 1,
x2 = 4.

With simple calculation we know x3 must be 1 for the sample mean to become 2. In another word, when first and second variables are determined, the third variable must be specific value(not free to be any value) to meet the fact that sample mean must be 2.

Ok, until this point everything makes sense to me.

When it comes to calculating the variance, I am totally confused. In each and every videos and articles I watched and read always says the same thing. The reason why degree of freedom for estimating the variance is N-1, which N is the sample size. The reason for that is because we know the sample mean is going to equal to specific value in the first place. This logic sound quite contradicting to me!!

The reason why we need a sample mean instead of the population mean is because we do not know the value of population mean in the first place. Now you are telling me that we know the sample mean is going to be specific value as if we already know the population mean? In another word, if we don't know what the population mean is, how are we possibly going to know what value our sample mean is even close to?

Does not the entire concept just sound contradicting? Maybe it is just because of my lack of knowledge but if that is the case, can anyone help me out by explaining to me the concept in simple English?

To summarise the question:

I do not understand why in the first example, we want to assume the sample mean is going to be 2, since in reality when we draw a sample from a population we don't even know the population mean, so how are we going to know sample mean is going to be 2?

---

@ImATurtle, Thanks for your detailed answer. In response to your answer, I understood that knowing the sample mean is necessary for calculating sample variance. However, my problem is I don't really get how you are going to know the sample mean before you even know all your sample values in the first place. For example, given sample values are {2,3,4,5,6}, we know that the sample mean is 4 only after you draw the sample and do the calculation . The five sample values were freely drawn from the population without doubt. Yet, people keep on saying that we know the sample mean will be 4, therefore after drawing the sample values 2,3,4,5, we know the fifth value is 6. So, what make it possible for us to predict the sample mean is going to be 4 before we even start drawing samples from population?

Answer 1

Let $(X_1,\ldots,X_n)$ be a simple random sample from normal distribution $\mathcal{N}(\mu, \sigma^2)$ with mean $\mu$ and variance $\sigma^2$. We can write this vector as $$ (X_1, \ldots, X_n) = \bar{X}_n (1, \ldots, 1) + (X_1 - \bar{X}_n, \ldots, X_n - \bar{X}_n) . $$ The first vector in this sum belongs to $1$-dimensional vector space $H =\{\lambda (1, \ldots, 1) | \lambda \in \mathbb{R}\}$ spanned by the vector $(1, \ldots, 1)$ and the second vector belongs to $(n - 1)$-dimensional vector space $H^{\bot} =\{(x_1, \ldots, x_n) \in \mathbb{R}^n | x_1 + \cdots + x_n = 0\}$ (you can check that the sum $X_1 + \cdots + X_n - n \bar{X}_n = 0$). Vector spaces $H$ and $H^{\bot}$ are orthogonal in the sense that for each $u \in H$ and $v \in H^{\bot}$ $$\langle u, v \rangle = 0$$ where $\langle (u_1, \ldots, u_n), (v_1, \ldots, v_n) \rangle = u_1 v_1 + \cdots + u_n v_n$ is the standard scalar product on $\mathbb{R}^n$, and vectors $\bar{X}_n (1, \ldots, 1)$ and $(X_1 - \bar{X}_n, \ldots, X_n - \bar{X}_n)$ are called the orthogonal projections of vector $(X_1, \ldots, X_n)$ on vector spaces $H$ and $H^{\bot}$.

We estimate the variance $\sigma^2$ by the sample variance $$S_n^2 = \frac{1}{n} \sum_{i = 1}^n (X_i - \bar{X}_n)^2 $$ that is, the sum of squares of coordinates of random vector $(X_1 - \bar{X}_n, \ldots, X_n - \bar{X}_n)$ we already encountered as an orthogonal projection of vector $(X_1, \ldots, X_n)$ on the $(n - 1)$-dimensional vector space $H^{\bot}$ and degrees you ask about corresponds to dimension of vector space $H^{\bot}$, since then, by Cochran's theorem, $S_n^2$ has $\chi^2$ distribution with $n - 1$ degrees of freedom.

You could say that, when you estimate parameters $(\mu, \sigma^2)$ of normal distribution with $(X_n, S_n^2)$ of $n$ degrees of freedom in the sample, $1$ goes into estimation of $\mu$ and remaining $n - 1$ into estimation of $\sigma^2$.

Answer 2

My understanding is that if we observe the statistic for the sample variance as being

$$s^2= \frac {\sum_{i=1}^{n} (y_i - \bar y)^2}{n-1}$$

because it is an $\mathbf{unbiased}$ $\mathbf{estimator}$ for the variance. Unbiased simply means that with repeated trials, we observe a convergence under probability of $s^2$ to $\sigma^2$; as $n \rightarrow \infty$, the probability that $s^2=\sigma^2$ is equal to 1. This follows all follows from that fact that

$$\frac{(n-1)s^2}{\sigma^2} \sim \chi^2 (\mathbf{df}=n-1)$$

(And also the fact that the $\chi^2$ distribution is just a special case of the Gamma distribution)

For a less theoretical explanation, I can tell you that you do need to know the sample mean in order to be able to calculate the variance; you cannot calculate deviance from the mean without knowing what the mean is. The same data used to calculate the mean is used to calculate the variance.. So the sample mean is inherently "fixed within the data" if you'd like.

Maybe I've just not fully understood your question. I'd be happy to elaborate more on specific parts if you wish.

Answer 3

This is late but I'm wondering if I understand things correctly as well. The way I see it, the sample mean is obtained from other sample means and is used to predict the population mean and variance, as well as the mean and variance for other samples.

Anyway, I think that it's N-1 because a sample of size 1 has a mean, itself, but doesn't have a variance. For chi-square, which gives the distribution of (the sum of the squares of the distance from the mean) of N standard normal random variables, will need to sum N-1, or V, squares. This gives the distribution of the distance for each random variable to the true mean. The t distribution is the normal distribution for the whole population divided by the square root of the average (ratio of the sample variance to the true variance), that had N-1 degrees of freedom. The distribution of the whole population divided by the ratio of the standard deviation of the sample to the standard deviation of the whole. This changes the variance of the t distribution according to degrees of freedom for the sample standard distribution.