Can I get weighted sample variance from the individual variances?

Question

Suppose I have daily aggregates: mean ($m_i$), variance ($v_i$) and number of samples ($n_i$); and I want to calculate weekly aggregates from these.

Weekly mean, $M$, is simple: its just a weighted mean. $$M = \frac{\sum n_i * m_i} {\sum n_i} $$

How about weekly variance, $V$? Can I just calculate $V$ like this?

$$ V = \frac{\sum n_i * v_i} {\sum n_i} $$

There are formulas on wikipedia that let you get the weighted sample variance if you have access to each sample, but I don't have access to all the samples over the whole week (well, I do have access, but a lot of data if I want to do monthly aggregates, for example). If needed, I can get any other daily aggregate values though.

Thanks!

Answer 1

The answer is not quite as simple as that. You are correct that if the $i$-th data set consists of $n_i$ samples with mean $m_i$, then the mean of the aggregated data set is $$M = \frac{\sum_i n_i m_i}{\sum_i n_i} = \frac{1}{N}\sum_i n_i m_i\tag{1}$$ where $N$ is the sum of all the $n_i$. Variances are a little trickier. The variance of the $i$-th data set is $$\begin{align} v_i &= \frac{1}{n_i-1}\sum_{j=1}^{n_i} \left[ \left(y_j^{(i)}\right)^2 - m_i^2\right]\tag{2}\\ &= \left[\frac{1}{n_i-1} \sum_j \left(y_j^{(i)}\right)^2\right ] - \frac{n_im_i^2}{n_i-1}\tag{3} \end{align}$$ where $y_j^{(i)}$ is the value of the $j$-th sample in the $i$-th data set. So, from $(2)$, we can get the value of the sum of the squared sample values as $$\sum_{j=1}^{n_i} \left(y_j^{(i)}\right)^2 = (n_i-1)v_i + n_im_i^2.\tag{4}$$ Thus, the sum of the squares of all the $N$ samples in the aggregate data set is $$\sum_i \sum_{j=1}^{n_i} \left(y_j^{(i)}\right)^2 = \sum_i (n_i-1)v_i + n_im_i^2. \tag{5}$$ Now, the aggregate data set has $N$ samples with mean $M$ as given in $(1)$. The standard formula for the variance of a data set is a generic version of $(3)$ and so we can write the variance of the aggregate data set as $$\begin{align} V &= \frac{1}{N-1}\sum \text{squares of all the sample values} - \frac{N}{N-1}\text{square of mean}\\ &= \left [ \frac{1}{N-1} \sum_i (n_i-1)v_i + n_im_i^2\ \right ] - \frac{NM^2}{N-1} \end{align}$$ where we used the result in $(5)$ for the sum of the squares of all the $N$ sample values.