4

I am reading about the Likelihood Principle (https://en.wikipedia.org/wiki/Likelihood_principle).

In short, I think the Likelihood Principle discusses the phenomena that if two experiments have the same underlying likelihood function, then even if these experiments result in different observed data - the parameter estimates will be the same in both experiments ... regardless of the fact that both experiments produced different data. That is, if two likelihoods are identical and only differ by some constant: optimizing these two likelihoods will give you the same parameter estimates. I am trying to understand why the Likelihood Principle is important.

Part 1: Here is an example from the Wikipedia Page:

Suppose we have two experiments involving independent Bernoulli trials with a probability of success on each trial given by $\theta$. In both experiments, we are interested in estimating $\theta$ based on the data we observe.

The likelihood functions for these two experiments are given by:

For $X = 3$, the likelihood function is:

$$\mathcal{L}(\theta \mid X=3) = \binom{12}{3} \theta^3 (1-\theta)^9 = 220\theta^3(1-\theta)^9$$ For $Y = 12$, the likelihood function is:

$$\mathcal{L}(\theta \mid Y=12) = \binom{11}{2} \theta^3 (1-\theta)^9 = 55 \theta^3 (1-\theta)^9$$

We can see that one of these likelihoods is 4 times the other likelihood. This means that both likelihoods are essentially identical and only differ by a constant term.

Part 2: I tried to continue these examples by myself:

For $X = 3$, the likelihood function is: $\mathcal{L}(\theta \mid X=3) = 220 \theta^3 (1-\theta)^9$

For $Y = 12$, the likelihood function is: $\mathcal{L}(\theta \mid Y=12) = 55 \theta^3 (1-\theta)^9$

To find the MLE, take the derivative of the likelihood function with respect to $\theta$, set it equal to zero, and solve for $\theta$. For both likelihood functions, the derivative is (not that the constant terms in both likelihoods 220 and 55 would cancel out when you would try to solve for $\theta$):

$$\frac{d}{d\theta} \mathcal{L}(\theta) = 3\theta^2(1-\theta)^9 - 9\theta^3(1-\theta)^8$$ Setting this equal to zero gives:

$$3\theta^2(1-\theta)^9 - 9\theta^3(1-\theta)^8 = 0$$

We can factor out $\theta^2(1-\theta)^8$ from both terms:

$$\theta^2(1-\theta)^8 (3(1-\theta) - 9\theta) = 0$$

Setting each factor equal to zero gives the possible solutions for $\theta$:

  • $\theta^2 = 0 \Rightarrow \theta = 0$
  • $(1-\theta)^8 = 0 \Rightarrow \theta = 1$
  • $3(1-\theta) - 9\theta = 0 \Rightarrow \theta = \frac{1}{4}$

The possible solutions for $\theta$ are 0, 1, and 1/4. However, $\theta=0$ and $\theta=1$ are likely not valid solutions because they imply that the event (observing a head) is either impossible or certain to happen. Therefore, the only likely valid solution is $\theta=1/4$.

Thus, we can see that both likelihoods produce the same estimate for $\theta$, thus demonstrating the Likelihood Principle.

My Question: But why is the Likelihood Principle important? The above exercise just showed me that the same likelihood function produces the same parameter estimates in different situations - Is this problematic? Is this useful? What is the big deal here? What is the big deal about the Likelihood Principle?

I have a feeling that the Likelihood Principle is trying to highlight some flaw about Likelihood theory - perhaps something about the uniqueness of parameter estimates from Maximum Likelihood Estimation ... or that the design of experiments is more important than the data collected from the experiments. But I am not sure.

Can someone please help me understand why the Likelihood Principle is important?

Thanks!

stats_noob
  • 3,112
  • 4
  • 10
  • 36

2 Answers2

2

What's important about the likelihood principle is that, under a particular assumed model for a (partially) unknown distribution (e.g., a probability model with one or more unknown parameters), all of the important information about the model that is contained in the data as a whole is actually contained within a single function, the likelihood, calculated from that data.

Back up for a moment, and imagine trying to measure the fairness of a coin through a series of 1000 independent flips. You write them all down, and it looks like this:

TTHTTHTTTTTHHHTHTTTHTHHHTHTHHHTTHTTHTHTHHHTHHHTTHHTTTTHHTTTHTTTHTHTHTTHHHHHTTTTHHTTHTHHHTTHHTTHTHHTTTHHHTTTTHTTTHTHHTTHHTHTHHHHHTTTHTHHHHHHHHHHTTHTHTTTTTHHHTTHHHTTHTTHHTTHHTTTTTTTHHHTHHTHTTTTTHTHHTHHHTHTTHHHTHTHHTTTHHHHHHTHHTHHTTHTTHHHTHTHTTHHTHTHHTTTHTTHHHTHHTHHTTTHHHHTHHTHTHTHTTHHHHTTHHHHHTHHHHTTTHHHHTHTHTTHTHHTHHTTTTHHHTTTHHTHTHTTTHTHHHTTTHTTHHTTTTTHTTTHHTHHHHHTTHTTHTHTHTHTTHTTHHTTTTHHTHTTHHTTTTHHTTHHTTHHTTHHTTTHHHHTTTTTTHHHHTTTHTTTHTHTHHTTHHHHHHTTTTHHHHHTHHHHTTTTTHHTTHTTTHTHTTTTTTHTTHHHTHHTTHHTHTTHHHTTTHHTTHHTHTHTTTTTTTTHTHTHHTTTHTHTTHHHHHHHTTTTTTHTTTTHHTHHTHTTHTHHTTHHHTHTTHTTHTTHHTTHHHTTTTHHTTTTHTTTTTHTHHTHHTHHHHHHHTTTHTTHTHTHTTHHTTTHHHTHTTHTTHHHTHHHHHTTTTHHHHHTTTTHHHTHTTHHTTHTTTTHHHHTTTTHTHHTHTTHHTTTTTHTHHHTHTHTHHHHTTHTTHTHHHHHTTTTTHHTTHHTHTTTHTHTTHTTTTHHTHTTTTTHHHTTHHHTHHHTHTTHHTTHHHTTHTTTTTHTHHTTTTTHHHHTTTHHTHTHTTHHHTTHTTHTHTTHHTTTHHHTTTTTHHHHTTTHHTHHHHHHHHTTHTTHHHHHTHHHTTHHTTHTTHHHHHTHTTHHTTTHTHHHTHHHTHHHHHTTHTTHTHHTTTHTHTTTTTTTTTHTHHHHHHHHHHHTTTHHTTTHTTTTTTHHTTHHTHHTHTHTTHTTT

There are lots of interesting patterns here. There's that huge run of tails followed by a huge run of heads in the last line. If you back away from the screen and squint your eyes, you can see that the different characters create strange blobs that don't seem entirely random. It's perhaps a little unclear which aspects of these data are relevant to determining how fair this coin is. I mean, clearly the total number of heads (492) and tails (508) are relevant, and it seems unlikely that a bunch of abstract "blobs" are important, but is it maybe useful to know that the number of runs of six or more heads (7) is the same as the number of runs of six or more tails (also 7)?

If we assume a particular model of the random process that created this list of head and tails (for example that the data correspond to 1000 independent and identically distributed flips with probability of H and T equal to $p$ and $1-p$ respectively), we can calculate the likelihood of the parameter $p$ based on these data. To do it the tedious way, instead of leaping right to the binomial formula, the likelihood of $p$ based on first data item alone is: $$L(p | \text{first flip is T}) = 1-p$$ The likelihood of $p$ based on the second and third data items alone is: $$L(p | \text{second flip is T}) = 1-p$$ $$L(p | \text{third flip is H}) = p$$ and so on. Likelihoods based on independent chunks of data can be multiplied to calculate an overall likelihood, and so the likelihood for $p$ based on all 1000 flips will be: $$L(p|\text{all data}) = (1-p)\times(1-p)\times p \times (1-p) \times \dots \times (1-p)$$ This obscures the dependency of $L$ on the actual data, so let's instead write down the likelihood as a function of arbitrary data. If we code our data as $x_i, 1\leq i\leq 1000$ with $x_i=1$ if the $i$th flip is a head and $x_i=0$ if it's a tail, the likelihood can be written: $$L(p|\mathbf{x}) = \prod_{i=1}^{1000} p^{x_i} (1-p)^{1-x_i}$$

This can be simplified. If we write $n=\sum_{i=1}^{1000} x_i$ as the total number of heads, this product is equal to: $$L(p|\mathbf{x}) = p^n(1-p)^{1000-n}$$

Maybe you're unimpressed, but I think something remarkable has happened here. Without explicitly appealing to the binomial distribution or starting from an assumption that the important part of these data was the total number of heads, we have just DERIVED the fact that all of the important information about the parameter $p$ in these data (i.e., the likelihood function) depends only on the number of heads in the data. So, the "blobs" aren't important; the number of long runs of heads and tails aren't important; nothing is important except the total number of heads. We can throw away our original data and keep a single number $n=492$, and this contains all the useful information for estimating $p$, performing a hypothesis test of $H_0\colon p=1/2$, determining which of two possible $p$s was most likely to have generated these data, etc.

Note that what's important depends on the assumed model (of course, since the likelihood is calculated from the model). If we instead assumed that the process generating these flips involved starting with a random, fair flip but then generating each subsequent flip by changing it with probability $p$ or leaving it the same with probability $1-p$, then our likelihood would be: $$L(p|\mathbf{x}) = \frac 12 \prod_{i=1}^{999} p^{|x_{i+1}-x_i|}(1-p)^{1-|x_{i+1}-x_i|}$$ The constant $1/2$ here comes from the first, fair flip, which contributes a likelihood of $1/2$ whether its heads or tails. (Since it's just a constant, we could throw it out, but I've kept it.) The rest of the likelihood depends on the data only through the expressions $|x_{i+1}-x_i|$ which take on a value 1 every time two consecutive flips are unequal. So, if we define $c$ as the total number of changes between consecutive flips (out of 999 consecutive pairs), the likelihood is: $$L(p|\mathbf{X}) = \frac12 p^c(1-p)^{(999-c)}$$ So, for this model, estimation of the parameter $p$ depends on the data only through the number of dissimilar consecutive pairs of flips, not on the starting flip or the total number of heads or tails, or the number of runs of six or more heads, or whatever.

In these simple examples, maybe its obvious. If you want to know if a coin is fair, maybe its obvious that you just need to count the total number of heads without worrying about the patterns of their occurrence. If you want to know if a coin "flips poorly" with subsequent flips depending on previous flips, maybe its obvious that you just need to count the number of consecutive differences.

Less obviously, the likelihood principle can be used to show that the only relevant information in a random sample from a normal distribution with unknown mean and variance is the sum of the observations and the sum of the squared observations. This follows from the likelihood: $$L(\mu,\theta) =\prod_{i=1}^n \frac{1}{\sqrt{2\pi}\sigma}\exp\left(-\frac12\frac{(x_i-\mu)^2}{\sigma^2}\right) = \frac 1{(2\pi\sigma^2)^{n/2}}\exp\left(-\frac12 \frac{\sum x_i^2 - 2\mu\sum x_i + n\mu^2}{\sigma^2}\right) $$ which depends on the data only through $\sum x_i$ and $\sum x_i^2$. So, feel free to throw your Gaussian data away, and just keep a couple of sums around!

(Note that, through some rearrangement of terms, you could show that the likelihood depends on the data only through the sample mean $\bar{x}$ and sample variance $s^2$, which is the more traditional set of so-called "sufficient statistics".)

K. A. Buhr
  • 1,672
1

There is an interesting discussion in "Statistical Inference", second edition, 2002, by George Casella and Roger Berger on pages 293-296 about what they call the formal versions of the likelihood, sufficiency and conditionality principles (henceforth LP, SP and CP respectively - see the end of this answer for the definitions). This suggests they also have informal counterparts, which they do, but these are just different in the sense of being less detailed, rather than lacking formality. What follows is essentially a summary of the thoughts of Casella and Berger (henceforth CB02) with a few of my own opinions thrown in for good measure. I am not expert in this so any mistakes are my own and not CB02, and I look forward to other people's thoughts on these matters.

Essentially it seems to me that CB02 are concluding there is some sort of "disconnect" between the popularity of discussing these principles and the reality of data analysis in practice whereby many statisticians do things that violate one or more of them - for instance CB02 point to the fact that the common practice of analysing model residuals are not based upon sufficient statistics and hence violate the SP. A key result is Birnbaum's theorem which states the LP implies and is implied by both the CP and SP being true. Thus violating either or both of the CP and SP violate the LP. Armed with this result we can see that residual analyses also violate the LP (CB02 also state residual analyses directly violate the LP - presumably by ignoring completely the likelihood function, but they did not elaborate).

The OPs question was interesting to me because I too have often thought "what is the big deal" about the LP, but from the perspective of it being some guiding principle we must adhere to - hitherto being unaware of the CB02 discussion which appears and old and well told tale. The point made by CB02 that is clearest to me when looking at the three principles is that they are all model-dependent. In this regard for the SP CB02 state "One shortcoming of this principle, one that invites violation, is that it is very model-dependent" and that "...belief in this principle necessities belief in the model, something that may not be easy to do". The main story is told by CB02 with Birnbaum's theorem taking center stage as the glue that holds all three principles together, but used in the opposite direction of the residual analyses example: if we do something that violates the LP then one or both of the CP and SP are violated. To my mind what is implied by CB02 but not explicitly stated is that this causes a philosophical problem if one accepts that the CP and SP are sensible principles to adhere to (i.e. if one believes the model).

One interesting argument put forward by CB02 to explain the willingness of statistical practice to violate the LP is not philosophical but instead technical: they state the proof of the LP following from the CP and SP in Birnbaum's theorem "is not convincing". They go on to explain that the proof invokes the SP before it invokes the CP, thus permitting a single sufficient statistic for the "mixture experiment" to be used, which leads to the result. They conclude by stating "At any rate, since many intuitively appealing inference procedures do violate the LP, it is not universally accepted by all statisticians. Yet it is mathematically appealing and does suggest a useful data reduction technique".

#################################################

Define an experiment $E$ to be the triple $E=\{\textbf{X},\theta,f(x|\theta)\}$ where $\textbf{X}$ is a random vector having the density $f(\textbf{x}|\theta)$ for some $\theta\in \Theta$. For any observed $\textbf{X}=\textbf{x}$ they go on to define $\text{Ev}(E,\textbf{x})$ to mean the evidence about $\theta$ arising from $E$ and $\theta$. This "evidence function" is intentionally left unspecified, even in formal proofs, but I guess is supposed to mean a statistic arising from the observed data, but perhaps also other concepts as well.

Formal likelihood principle: Suppose $E_{1}=\{\textbf{X}_{1},\theta,f_{1}(x_{1}|\theta)\}$ and $E_{2}=\{\textbf{X}_{2},\theta,f_{2}(x_{2}|\theta)\}$ are two experiments where the unknown $\theta$ is the same in both. Suppose $\textbf{x}^{*}_{1}$ and $\textbf{x}^{*}_{2}$ are two sample points from $E_{1}$ and $E_{2}$ respectively such that for all $\theta\in\Theta$

\begin{align*} L(\theta|\textbf{x}^{*}_{2})=CL(\theta|\textbf{x}^{*}_{1}) \end{align*}

for some constant $C$ that may depend on $\textbf{x}^{*}_{1}$ and/or $\textbf{x}^{*}_{2}$ but not on $\theta$. Then;

\begin{align*} \text{Ev}(E_{1},\textbf{x}^{*}_{1})=\text{Ev}(E_{2},\textbf{x}^{*}_{2}) \end{align*}

This contrasts with their informal definition on page 291;

Informal likelihood principle: Let $f(\textbf{x}|\theta)$ be the joint pdf or pmf of the sample $\textbf{X}=(x_{1},...,x_{n})$. If $\textbf{x}$ and $\textbf{y}$ are two sample points such that for all $\theta\in\Theta$

\begin{align*} L(\theta|\textbf{x})=C(\textbf{x},\textbf{y})L(\theta|\textbf{y}) \end{align*}

then the conclusions drawn from $\textbf{x}$ and $\textbf{y}$ should be identical.

As described on page 294 they state the difference between the two principles are that the formal principle has two experiments whereas the informal has only one . They point out that setting $E_{1}=E_{2}=E$ in the formal principle gives the informal principle. In contrast to these likelihood principles the following formal and informal versions of the sufficiency principle are more intrinsically different rather than just the formal being stronger than the informal (not sure if this is the best use of language here).

Formal sufficiency principle: Let $E=\{\textbf{X},\theta,f(x|\theta)\}$ be an experiment and suppose $T(\textbf{X})$ is a sufficient statistic for $\theta$. If $\textbf{x}$ and $\textbf{y}$ are two sample points satisfying $T(\textbf{X})=T(\textbf{Y})$ then $\text{Ev}(E,\textbf{X})=\text{Ev}(E,\textbf{Y})$

This contrasts with their informal definition on page 279;

Informal sufficiency principle: Let $X_{1},....,X_{n}$ be $iid$ observations from a pdf or pmf $f(x|\bf{\theta})$ that belongs to an exponential family given by\

\begin{align*} f(x|\bf{\theta})=h(x)c(\bf{\theta})\exp\left(\sum_{i=1}^{k}w_{i}(\bf{\theta})t_{i}(x)\right) \end{align*}

where $\bf{\theta}\in\mathbb{R}^{d}$ for $d\leq k$. Then;

\begin{align*} T(\textbf{X})=\left(\sum_{j=1}^{n}t_{1}(X_{j}),...,\sum_{j=1}^{n}t_{k}(X_{j})\right) \end{align*}

is a sufficient statistic for $\bf{\theta}$

On page 293 of CB02 they describe that the formal SP goes further than the informal one (which does not mention any experiment) by "agreeing to equate evidence if the sufficient statistics match"

Conditionality principle: Suppose $E_{1}=\{\textbf{X}_{1},\theta,f_{1}(\textbf{x}_{1}|\theta)\}$ and $E_{2}=\{\textbf{X}_{2},\theta,f_{2}(\textbf{x}_{2}|\theta)\}$ are two experiments where only the unknown parameter $\theta$ need be common between the two experiments. Consider the mixed experiment where the random variable $J$ is observed where $P(J=1)=P(J=2)=0.5$ (independent of $\theta$, $\textbf{X}_{1}$ or $\textbf{X}_{2}$), and then the experiment $E_{J}$ is performed. Formally the experiment performed is $E^{*}=\{\textbf{X}^{*},\theta,f^{*}(\textbf{x}^{*}|\theta)\}$ where $\textbf{X}^{*}=(j,\textbf{X}_{j})$ and $f^{*}(\textbf{x}^{*}|\theta)=f^{*}[(j,\textbf{x}_{j})|\theta]=0.5f_{j}(\textbf{x}_{j}|\theta)$. Then

\begin{align*} \text{Ev}(E^{*},(j,\textbf{x}_{j}))=\text{Ev}(E_{j},(j,\textbf{x}_{j})) \end{align*}

dandar
  • 980