It was mentioned in pLSA paper that perplexity refers to the log-averaged inverse probability on unseen data. Can any one give me the exact formula for calculating perplexity
Asked
Active
Viewed 4,249 times
1
-
Searching Google for "log-average" yields as the first hit the Wikipedia article for geometric mean, with a conveniently highlighted excerpt stating that it is also sometimes called the log-average... – May 05 '11 at 04:33
-
@Rahul: I too found the wikipedia article, but im not convinced with that definition in the context of pLSA probabilities. Looking for better information on the same. – Learner May 05 '11 at 04:36
-
1Also at http://stats.stackexchange.com/questions/10302/what-is-perplexity – Henry May 05 '11 at 06:43
1 Answers
3
You have looked at the Wikipedia article on perplexity. It gives the perplexity of a discrete distribution as
$$2^{-\sum_x p(x)\log_2 p(x)}$$
which could also be written as
$$\exp\left({\sum_x p(x)\log_e \frac{1}{p(x)}}\right)$$
i.e. as a weighted geometric average of the inverses of the probabilities. For a continuous distribution, the sum would turn into a integral.
The article also gives a way of estimating perplexity for a model using $N$ pieces of test data
$$2^{-\sum_{i=1}^N \frac{1}{N} \log_2 q(x_i)}$$
which could also be written
$$\exp\left(\frac{{\sum_{i=1}^N \log_e \left(\dfrac{1}{q(x_i)}\right)}}{N}\right) \text{ or } \sqrt[N]{\prod_{i=1}^N \frac{1}{q(x_i)}}$$
or in a variety of other ways, and this should make it even clearer where "log-average inverse probability" comes from.
Henry
- 157,058
-
can the discrete form of perplexity shown here be used to measure the extent of a distribution? or is that reserved for the integral version of perplexity (exponential differential entropy)? – develarist Sep 29 '20 at 04:08
-
@develarist That rather depends what you mean by "extent". Perhaps it might be described as a measurement of something like "diversity" – Henry Sep 29 '20 at 08:00