Mathematics Stack Exchange
Mathematics Stack Exchange

Questions tagged [entropy]

This tag is for questions about mathematical entropy. If you have a question about thermodynamical entropy, visit Physics Stack Exchange or Chemistry Stack Exchange instead.

This tag is for questions about mathematical entropy, not to be confused with thermodynamical entropy which goes in Physics or Chemistry Stack Exchange.

1617 questions
1
vote
0 answers

Value function of deep energy-based policies

I am reading the following paper: https://arxiv.org/pdf/1702.08165.pdf I am stuck with one of their proofs, more precisely equation (14): $$ H(\pi(\cdot|s))) + \mathbb{E}_{a \sim b}[Q_{soft}^{\pi}(s,a)] = -D_{KL}(\pi(\cdot |s) || \tilde{\pi}(\cdot…
Andreas Pasternak
  • 111
1
vote
1 answer

Word separator entropy in passphrases calculation

I'm trying to find out how much entropy word separators add to a passphrase. Let's say the word list from which we generate our passphrase is 1000 words. If we add space as word separator, how much does the entropy change? what if we add 2@f as…
oatmealisgood
  • 11
  • 1
1
vote
1 answer

Equality of entropy $\iff$ Same probabilities under permutation?

Assume I have: $$H(p_1,\ldots,p_N)=H(q_1,\ldots,q_N)$$ where $H$ is the Shannon Entropy. Does that mean that I necessarily have the $p_i$ and $q_i$ linked by a permutation? Or is it not true? For the case with $N=2$ I know it is true but what for a…
StarBucK
  • 689
1
vote
0 answers

Entropy of the error function

Can anyone tell me the following indefinite integral (antiderivative): $$ \int \frac{a}{2} \left( 1 + \textrm{erf}(x) \right) \log\left( \frac{a}{2}\left(1 + \textrm{erf}(x) \right) \right) dx $$ $$ = \int \frac{a}{2} \textrm{erfc}(-x) \log\left(…
Coffee
  • 21
  • 4
1
vote
1 answer

Entropy contribution from variable length segment of a sequence.

If I have a sequence which is comprised of one of $10$ prefixes, one of $5$ suffixes and a variable length middle, how do I compute the entropy of the sequence? Using Shannon-Entropy $$H= -\sum_{i=1}^{m} p_i \ln(p_i)$$ I can compute the entropy…
Nick
  • 185
1
vote
0 answers

Need help calculating entropy using huge kronecker products

I am no expert in tensor algebra. I am stuck with computing the following function of kronecker products (you will recognize an entropy-like equation): $H(\mathbf{T}) = \mathbf{T}^t \cdot \log_2(\mathbf{T})$ where $\mathbf{T}'$ is the transpose of…
Cesare
  • 346
1
vote
1 answer

Simplyfying Shannon Entropy formula

I need to calculate the following which is Shannon Entropy formula only using simple functions like log or squareroot or I don't know, just make it simple enough for a guy that does understand only grammar school math, if that is possible, so I…
Vlastimil Burián
  • 125
1
vote
1 answer

Not understanding steps in derivation for entropy of a Gaussian random variable

Can someone explain the last two steps in the derivation given below? This is the derivation of the entropy of a Gaussian random variable:
user2233120
  • 23
1
vote
0 answers

Entropy of first char of a word in dictionary

Suppose I have an English dictionary, that is a list of words in a file. I have to calculate the entropy of the first char. I calculated the probability of each first char ($P_a, P_b, \dots, P_z$) in this way: $$P_a = \dfrac {\# \text{words that…
Katherine Maurus
  • 199
  • 1
  • 9
1
vote
0 answers

How can I determine the upper limit of Shannon Entropy?

I know that the maximum possible Shannon Entropy for an alphabet $X$ is $\log|X|$, where Shannon Entropy is: $$H(X) = - \sum_{x \in X} \; p(x) \log p(x)$$ but how is this upper limit computed?
brabster
  • 111
1
vote
1 answer

Is there a connection between topological entropy and stationary distributions?

In a book I read the following: "The topological entropy is the supremum over all stationary distributions of the entropy of the corresponding stationary sequence." I did not find this definition anywhere else, do you know this definition?
Salamo
  • 1,094
1
vote
0 answers

Integrating entropy on an arbitrary boundary

Entropy, denoted as H, is: $$ H = -\int_a^b p\ln(p) dx $$ where the range a to b is some arbitrary boundary and where p is given by the classic: $$ p(x) = \frac{1}{\sigma\sqrt{2\pi}}e^{-\frac{1}{2}\left( \frac{x-\mu}{\sigma}\right)^2} $$ Here is…
warship
  • 162
1
vote
1 answer

What's the maximum entropy for discrete distribution given mean and variance

I know for continuous distribution, given mean and variance, it's Normal distribution. I wonder what the distribution or the maximum entropy would be if I constrain the mean and the variance. I assumed differential entropy would be a intelligent…
KH Kim
  • 225
1
vote
2 answers

For any $n\ge2$ prove that $H(X_1,X_2,...,X_n)\ge\sum\limits_{i=n}^\mathbb{n}\ H(X_i|X_j , j \neq i)$

I am trying to figure this out and I am stuck. Any ideas? For any $n\ge2$ prove that $H(X_1,X_2,\ldots,X_n)\ge\sum\limits_{i=1}^\mathbb{n}\ H(X_i\mid X_j , \ j \neq i)$
Wanderer
  • 927
1
vote
0 answers

Why is ln(p_i) not rounded down in theexpression for Shannon entropy?

Entropy supposedly " is the average amount of information contained in each message received"(Wikipedia: Entropy). However, to calculate the Shannon entropy for a finite sample, we have the sum over the of -p_i(log p_i). However, for each i, the…
nomadreid
  • 352
1 2
3
4 5 6