1

I'm reviewing my thin grasp of likelihood from years ago.

For simplicity, let us consider a finite set of candidate models for explaining a set of observed data. They can be completely different models or they can be a parameterized family of formulas for PDFs or probability mass functions (PMF). For parameterized models, this simple picture means that the parameters have been restricted to take on countably many values, even though we don't know what the values of the model parameters are.

As a simple example, consider the probability $p_H$ of heads for a coin. A friend fabricates a coin such that $p_H$ is one of eleven possible values: 0.0, 0.1, ... , 1.0. He doesn't tell me what $p_H$ is. Given a series of coin tosses, it is up to me to estimate $p_H$. Let's say I generate the likelihood function $L(p_H|O)$, where $p_H$ is one of 0.0, 0,1, ..., 1.0.

If I uniformly scale the likelihoods so that the values add to 1.0, it certainly looks like a PMF. The ratios between any two $p_H$ values make intuitive sense in reflecting how likely the two values are. In special cases like this, is the likelihood function also a probability function?

The reason I ask is because two reasons for likelihood to differ from probability are: (i) the set of models in my likelihood function might not cover all possible models that explain the observed data and (ii) the models might not be mutually exclusive. In this case. all possible values of $p_H$ are 0.0, 0.1, ..., 1.0 represent all possible models because my friend fabricated the coin with one such $p_H$ value. The models are also mutually exclusive, since (say) $p_H=0.3$ and $p_H=0.5$ are mutually exclusive.

2022-08-29 afternote: This answer says that scaled likelihood can be interpreted as probability when it happens to coincide with the posterior, and when the prior is a uniform distribution. I don't follow the math, but it seems that there is much debate. It is helpful, but it does require justification from a Bayesian perspective. Bayes aside, however, it doesn't answer the question of why a scaled likelihood function wouldn't be a probability distribution (i) if it covers all the possible models and (ii) in view of the meaningfulness of likelihood ratios -- quite apart from Bayes. Why are those two arguments alone insufficient?

user2153235
  • 189

1 Answers1

2

Here's a Bayesian answer to this question. Suppose that a priori we believe all models are equally likely, so that $P(p_H = x) = 1/11$ for $x = 0, 0.1, 0.2, \ldots, 1$. Then the posterior distribution of $p_H$ given the observations $O$ is $$ P(p_H = x | O) = \frac{P(O | p_H = x) P(p_H = x)}{p(O)} = \frac{P(O|p_H)P(p_H=x)}{\sum_y P(O|p_H=y)P(p_H=y)} = \frac{P(O|p_H)/11}{\sum_y P(O|p_H=y)/11} = \frac{P(O | p_H = x)}{\sum_{y} P(O | p_H = y)} $$ Note that $P(O | p_H = x)$ is the likelihood function evaluated at $p_H$, so the above quantity is the likelihood function that has been scaled to sum to $1$. So it can be interpreted as the posterior probability distribution of $p_H$ under a uniform prior distribution.

user2153235
  • 189
philbobaggins42
  • 61
  • Thanks. Shouldn't $P(p_H=x)=1/11$? What happened to $P(p_H=x)$ in the numerator? To avoid the prerequisite for a prior, can my question be answered without Bayesian theory, just based on the argument that the models are complete and that their relative likelihoods are meaningful? Finally, if those conditions are enough, is it generalizable beyond the coin toss example? – user2153235 Aug 28 '22 at 01:23
  • Should the denominator be $P(O)=\sum_y P(O|P_H=y)$? The variable $x$ can't be used as an iterator because it is the argument into the LHS. – user2153235 Aug 28 '22 at 03:23
  • Yes thanks for catching this. As for what happened to $P(p_H = x)$ in the numerator, it cancelled with the $p(p_H = y)$ in the denominator (use the law of total probability on $P(O)$). – philbobaggins42 Aug 28 '22 at 11:42
  • In the Frequentist setting, you should not think of likelihoods as probabilities. Because in this case the true $p_H$ is fixed (and not random). However, in this case the ratio between likelihoods is still useful for comparing the evidence for different models. See the likelihood ratio test. – philbobaggins42 Aug 28 '22 at 11:45
  • Thanks for the revisions. I added some detail to show the cancellations. As for not thinking of likelihoods as probabilities because a model parameter is fixed, I'm confused. Whenever I see $H_0$ for hypothesis testing, along with a (typically) bell shaped curve about the presumed mean, isn't that viewing the true population mean as distributed about the $H_0$ mean? That last sentence aside, doesn't the likelihood function have the key features of a probability mass function, i.e., the relative values meaningfully indicate how likely the true model is, and the probabilities sum to one? – user2153235 Aug 28 '22 at 16:48