When running a t-test with a Control and Experimental group, are both groups Samples? Or is the Control group a Population?

Question

When running a t-test with a Control and Experimental group, are both groups Samples? Or is the Control group a Population? I am pretty confused on this, since I've pretty much only ever worked with Samples, but have no idea when to switch to a population

Answer 1

If you use a 2-sample t test, then you're assuming data from Treatment and Control groups are both samples. In many cases the two samples are of about the same size. If you're designing an experiment, trying to know how much data to collect in order to have a reasonable chance of detecting a difference between the two groups, and you have a given number of subjects available, then it's most efficient to put half of the subjects (at random) into each group.

However, sometimes you have a situation in which huge control group already exists (say 100,000) and you imagine you can randomly take a smaller random sample (say 100) from the same population to use for a treatment group. Then it might not make much difference whether you do a two sample t test on the two groups, or you do a one-sample t test comparing the sample mean from the small control group against the mean of the huge group--regarded as a population mean.

Examples 1: Two small groups of about the same size.

Generate fake data for illustration:

set.seed(1234)
x.c = rnorm(100, 50, 5)
x.t = rnorm(100, 51, 5)

A boxplot show the treatment values (top) tend to be a little larger.

boxplot(x.c, x.t, col="skyblue2", horizontal=T)

A two-sample t test finds a significant difference (at the 5% level) between the mean $49.22$ of the control group and the mean $51.21$ of the treatment group because the P-value $0.0064 < 0.05 = 5\%.$

t.test(x.c, x.t)
    Welch Two Sample t-test


data:  x.c and x.t
t = -2.7635, df = 197.85, p-value = 0.006258
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 -3.4101030 -0.5699462
sample estimates:
mean of x mean of y 
 49.21619  51.20622 

Example 2: Huge control group and small treatment group.

set.seed(1235)
x.c = rnorm(100000, 50, 5)
x.t = rnorm(100, 51, 5)

A boxplot shows that the smaller control values (upper boxplot) tend to be larger. [It is typical for a huge normal sample to show many boxplot 'outliers'.]

 boxplot(x.c, x.t, col="skyblue2", horizontal=T)

A two-sample t test finds a significant difference (at the 5% level) between the mean $59.92$ of the control group and the mean $51.09$ of the treatment group because the P-value $0.036 < 0.05 = 5\%.$

t.test(x.c, x.t)
   Welch Two Sample t-test


data:  x.c and x.t
t = -2.1268, df = 99.195, p-value = 0.03592
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 -2.0685429 -0.0717558
sample estimates:
mean of x mean of y 
 50.01799  51.08814 

Example 3: Considering the huge control group as a population.

The mean of the control group is taken as $\mu = 50.02$ and we compare the sample mean of the control group to $\mu.$ In the boxplot below, the vertical line indicates the position of $\mu.$

boxplot(x.t, col="skyblue2", horizontal=T)
 abline(v = 50.02, col="red")

A one-sample t test finds a significant difference (at the 5% level) between the population mean $50.02$ of the control "population" and the sample mean $51.09$ of the smaller treatment group because the P-value is $0.036 < 0.05.$ Here we are ignoring any variability in the 'control' group, taking $\mu = 50.02$ as an established value. Even with a 'control population' as large as $100\,000$ this may be risky.

t.test(x.t, mu=50.02)   # Null population mean shown
One Sample t-test


data:  x.t
t = 2.1238, df = 99, p-value = 0.03618
alternative hypothesis: true mean is not equal to 50.02
95 percent confidence interval:
  50.09021 52.08607
sample estimates:
mean of x 
 51.08814