I'm looking for a generalization of the Bernoulli distribution to the continuous domain of [0, 1]. The main requirement is that $E[x] = \theta$.
Asked
Active
Viewed 656 times
1 Answers
0
An answer is contained in Theorem 5.1 of http://www.math.uconn.edu/~kconrad/blurbs/analysis/entropypost.pdf. Moreover, it is unnecessary to use variational calculus to prove the theorem. (Often people solve these problems with Lagrange multipliers but such methods feel like one-way arguments: if a solution exists using variational methods it must be such-and-such, with no proof that the optimal solution must be accessible to variational methods.)
KCd
- 46,062
-
@SherjilOzair, I am intrigued by your comment that this problem came up in the setting of machine learning. Could you point me to a place that explains why and how MaxEnt is used in this way? – KCd Mar 21 '17 at 22:25
-
The original comment was in reference to another thing. But MaxEnt is very important in machine learning, because it captures the idea of learning a model which informs itself by the data, but does not add any additional bias. The MaxEnt part is what ensures that you don't add any additional information, that the model is clueless when it's not informed by the data.
You may try, and links therein: https://en.wikipedia.org/wiki/Principle_of_maximum_entropy
Also: http://www.math.uconn.edu/~kconrad/blurbs/analysis/entropypost.pdf
– SherjilOzair Mar 21 '17 at 23:11 -
Your second link is exactly the one I had included in my answer. I know it rather well... – KCd Mar 22 '17 at 03:03
-
While this is probably the correct answer, it is a profound mystery to me why this distribution isn't translation invariant. Changing the interval from [a, b] to [a + e, b + e] changes the shape of the distribution.
If the only information we provide is that of a known mean, and bounds, surely that shouldn't the shape whether I'm talking about things of the order of 10s or 10,000s.
– SherjilOzair Mar 22 '17 at 03:24 -
I didn't notice your username. If you had written this article, I assume you already know most of what is there to be known on the subject of maxent and machine learning. – SherjilOzair Mar 22 '17 at 03:28
-
I know zilch about machine learning and the file I wrote says nothing on that topic, only about the principle of max. entropy. This is why I was intrigued by your earlier comment. – KCd Mar 22 '17 at 04:01
-
The mean of a distribution is affected by the numerical values of its density function, so it's not a surprise (to me) that a translation might affect things. Are you sure you mean to asking about additive rather than multiplicative translations? To illustrate what mystifies you by writing about "order of 10s or 10,000s" sounds like you are interested in changing an interval by multiplication rather than by addition. – KCd Mar 22 '17 at 04:05
Yes. The main reason beta doesn't work is because I do want the logpdf be a linear function in $\log (\theta)$ and $\log (1 - \theta)$. Beta also does not fulfill the analytic PDF, CDF, or ICDF requirements.
Machine Learning. This functional form is commonly used as a cost function to train binary classifiers. I want to generalize the cost function to regression in [0, 1] as well, while maintaining the same functional form, especially the forms for the gradient of the logpdf wrt $\theta$.
You might say I've changed the question dramatically. The new question is: what's the maximum entropy distribution on [0, 1] with a single mean parameter?
– SherjilOzair Mar 21 '17 at 21:33https://en.wikipedia.org/wiki/Maximum_entropy_probability_distribution#Examples
– SherjilOzair Mar 21 '17 at 21:43