1

https://www2.warwick.ac.uk/fac/cross_fac/complexity/study/msc_and_phd/co904/co904online/lecture-3-4.pdf (the without expectation setting) \begin{align*} maximize \ &\sum_{i=1}^K -p_i log(p_i) \\ \ subject \ to \ &\sum_{i=1}^K p_i =1 \end{align*} $p_i$ is the probability distribution over a set $\left\{x_i\right\}_{i=1}^K$.

I was just wondering: we have $\sum_{i=1}^N p_i =1$ (for a discrete probability distribution when we are trying to solve a maximum entropy problem) as a constraint. Is the correct reason as to why we don't need $p_i \ge 0 \ \forall i$ as another constraint that this is already captured as an implicit constraint in the $log(.)$ function (with a strict inequality, in fact)?

learning
  • 633

2 Answers2

1

The assumptions $p_i\geq 0$ must always be present, but depending on your setup it could be buried in some other assumptions. Can't say for sure unless you write out the full setup and optimization problem you are considering.

pre-kidney
  • 30,223
  • The problem would be maximize $\sum_{i=1}^K -p_i log(p_i)$ subject to $\sum_{i=1}^K p_i =1$. $p_i$ is the probability distribution over a set $\left{x_i\right}_{i=1}^K$. – learning Mar 21 '17 at 05:59
  • https://www2.warwick.ac.uk/fac/cross_fac/complexity/study/msc_and_phd/co904/co904online/lecture-3-4.pdf The one here. – learning Mar 21 '17 at 06:38
0

If $p_i<0$ for any $i$, then your objective function is undefined. Maximising an undefined function is also undefined. Under the assumption that your objective is defined (which seems reasonable), then you will only have solutions for $p_i \geq 0$. Note that for entropies we always define $0 \log 0 = 0$.

rey_pato
  • 11
  • 2