2

I've been reading a little about the Fisher-Neyman factorization theorem on my own, and I think I understand intuitively what a sufficient statistic is, but I am wondering what the convention is for formally defining it.

Is it true that, for a given collection of iid samples ($\vec{x}$) from a distribution $X$, the maximum likelihood estimate of the parameter $\theta \in \mathbb{R}^t$ depends exclusively on $T(\vec{x})$ and not on other factors such as the number of elements in the sample?

If that is the case, does it make sense to take the following as a definition of sufficiency. (Maximum likelihood estimates given $\vec{x}$ and $\vec{y}$ are equal iff $T(\vec{x}) = T(\vec{y})$).

$$ \forall { \vec{x}, \vec{y} \in \text{Seq}[\mathbb{R}]: } \left( \max_{\theta_x \in \mathbb{R}^t} \prod_{x \in \vec{x}} p(x;\theta_x) \right) = \left( \max_{\theta_y \in \mathbb{R}^t} \prod_{y \in \vec{y}} p(y;\theta_y) \right) \Longleftrightarrow T(\vec{x}) = T(\vec{y}) $$

So this definition, while closer to right, is also wrong.

Based on the comments below, the above definition inappropriately rejects all sufficient statistics except for the maximum likelihood estimate itself. For instance, the sum is a sufficient statistic for the mean in a normal distribution with standard deviation 1: $\mathcal{N}(\mu, 1)$. Amending the formula above gives us: $$ \forall { \vec{x}, \vec{y} \in \mathbb{R}^n: } \left( \max_{\theta_x \in \mathbb{R}^t} \prod_{x \in \vec{x}} p(x;\theta_x) \right) = \left( \max_{\theta_y \in \mathbb{R}^t} \prod_{y \in \vec{y}} p(y;\theta_y) \right) \Longleftrightarrow T(\vec{x}) = T(\vec{y}) $$

with $n$ being the length of the sample.

Greg Nisbet
  • 11,657
  • 1
    I think you are basically right. The same sufficient statistic will yield same estimator (assuming same sample sizes and parametric structures) due to the fact that the maximum likelihood estimator is a bijective function of the sufficient statistic. Regarding the intuition, such a statistic compresses the Fisher's information given in the joint distribution of the whole sample size, i.e., $\mathcal{I}{T(\mathrm{X})}(\theta) = \mathcal{I}{\mathrm{X}}(\theta)$, hence it is suffice to use it for (some) estimation purposes. – V. Vancak Dec 21 '15 at 13:30
  • @user230329, so does that mean the proposition I gave above is missing as a requirement that $\text{len(}\vec{x}) = \text{len}(\vec{y})$? I am trying to figure out how the sample size interacts is incorporated into the definition of a sufficient statistic, if at all. – Greg Nisbet Dec 21 '15 at 18:24
  • 1
    If you want to estimate the mean of an i.i.d. $N(\mu,1)$ sample, $T(\vec{x}) = \sum_i x_i$ is a sufficient statistic. But you stiill need to know something about the length of the vector to construct the estimate. Note that $\sum_i x_i / n$ is also a sufficient statistic, as is $\vec{x}$ itself (though obviously, the last one is not minimal). – Batman Dec 21 '15 at 20:53
  • 1
    @Batman: "You still need to know something about the length of the vector to construct the estimate": yes, that makes sense. Once you've constructed the estimate though is it the case that you no longer need the length? Otherwise, I'd be tempted to call $T(\vec{x}), \text{len}(\vec{x})$ the sufficient statistic ... – Greg Nisbet Dec 21 '15 at 23:10
  • Oh wait, I see what you are saying. The sum by itself is a sufficient statistic, even though you cannot use it to construct a maximum likelihood estimate without knowing the size of the sample. If you weren't allowed to use the length of the sample, then the only "sufficient statistic" would be the maximum likelihood estimate itself. – Greg Nisbet Dec 22 '15 at 00:18
  • 1
    I think in some cases the MLE is not sufficient, but no examples are coming to mind instantly. Maybe I'm rusty on this. – Michael Hardy Dec 22 '15 at 00:36

2 Answers2

2

Repeat after me:

"Sufficiency pertains to data reduction, not estimation."

I have said this on stats.SE as well. A sufficient statistic need not be related to an estimator for a parameter or parameters. A sufficient statistic only needs to not lose information about the parameter, relative to the information contained in the original sample. The original sample is sufficient (tautologically), yet it estimates nothing.

A "good" estimator is frequently based on a sufficient statistic, in the sense that a good estimator does not fail to use all of the information contained in the original sample to estimate the parameter(s) of interest. It is not a requirement (or even desirable), however, that a sufficient statistic be a "good" estimator.

heropup
  • 135,869
1

Suppose $(\Omega,\mathcal F,P)$ is a probability space for each $P$ in some family $\mathcal P$ of probability measures on $\mathcal F$. Then a random variable $T$ is sufficient for the family $\mathcal P$ if the conditional probability measure on $\Omega$ given the value of $T$ is the same for all $P\in\mathcal P$.

That's a bit abstract. In the situations considered in textbooks in which the concept of sufficiency is first introduced, you have an i.i.d. sequence $X_1,\ldots,X_n$ and a parametrized family of probability measures such as $N(\mu,\sigma^2)$. Then (in the case of that family of distributions) $T(X_1,\ldots,X_n)$ is a sufficient statistic for that family of distributions if the conditional distribution of $X_1,\ldots,X_n$ given $T(X_1,\ldots,X_n)$ is the same for all values of $\mu\in\mathbb R$ and $\sigma>0$. For this family of distributions the pair $(X_1+\cdots+X_n, X_1^2+\cdots+X_n^2)$ is sufficient. For the family of Gamma distributions $$ \frac 1 {\Gamma(\alpha)} \left( \frac x \theta \right)^{\alpha-1} e^{-x/\theta}\ \frac{dx}\theta \quad\text{for }x>0, $$ the pair $(X_1+\cdots+X_n, X_1\cdots X_n)$ (the sum and the product) is sufficient because the conditional distribution of $(X_1,\ldots,X_n)$ given the value of that pair does not depend on $(\alpha,\theta)$. For the family of uniform distributions on an interval $(\theta_1,\theta_2)$ the pair $(\min\{X_1,\ldots,X_n\}, \max(X_1,\ldots,X_n))$ is sufficient because the conditional distribution of $(X_1,\ldots,X_n)$ given $(\min,\max)$ is the same for all values of $(\theta_1,\theta_2)$ that are consistent with that value of $(\min,\max)$ ("consistent" here would mean $\theta_1\le\min$ and $\theta_2\ge\max$).

Michael Hardy
  • 1