Mathematics Stack Exchange
Mathematics Stack Exchange

Questions tagged [entropy]

This tag is for questions about mathematical entropy. If you have a question about thermodynamical entropy, visit Physics Stack Exchange or Chemistry Stack Exchange instead.

This tag is for questions about mathematical entropy, not to be confused with thermodynamical entropy which goes in Physics or Chemistry Stack Exchange.

1617 questions
0
votes
2 answers

How can I think of (visualize) 0.5 bits of information?

I was studying Shannon's Entropy function and for a 35% chance of a particular event, the formula produced the answer 1.5 bits. log2(0.35) = 1.5 bits(approx.) of information. I know it's pretty trivial in context of practical applications, but how…
Zaid Khan
  • 111
0
votes
0 answers

Prove $H(X) = H(Y)$

If I have $X$ and $Y$ discrete r.v.'s with outcomes $\{x_1,x_2\}$ and $\{y_1,y_2\}$ and entropies $H(X)$ and $H(Y)$. Let $P_{x|y}(x_1|y_1)=P_{x|y}(x_2,y_2)=0$. Prove that $H(X) = H(Y)$ (entropy theory). If I come up with example numbers and assign…
Jared
  • 75
  • 6
0
votes
1 answer

Differing Calculations? Entropy of ONE Alphabet $= 4.7$ or $0.18$?

If I choose a random alphabet from (a-z), $26$ characters, what is the entropy? Shannon's formula: $$H = - \sum p \log_2(p) = - (1/26)\log_2(1/26) = 0.18$$ bits. However, other formulas on the Internet use: $$H = \log_2 (N^L) = \log_2(26) =…
Bhagwad Jal Park
  • 113
0
votes
1 answer

When is the entropy H(X, Y) = H(X) = H(Y)?

I read in my textbook the following Corollary that follows from the chain rule for entropy: H(X, Y) >= H(X) or H(X, Y) >= H(Y) I was wondering what the necessary condition is for equality? If X and Y are random variables of course.
2000mroliver
  • 171
  • 2
  • 9
0
votes
0 answers

Binary Entropy Function (properties)

I'm not a mathematician and I'm trying to understand the properties of the binary entropy function. In particular, I would like to ask you, if the distributive law can be applied to the following expression: $(x_i - y_i) H(\frac{1}{2^{i-1}}) \quad…
Yiota
  • 1
0
votes
2 answers

Why the solution to this entropy problem is not simply $H(X)=-a*log(a) - (1-a)*log(1-a)$?

Let $X_1$ and $X_2$ be discrete random variables drawn according to probability mass function $p_1$ and $p_2$ over the respective alhabets $X_1={1,2,...m}$ and $X_2={m+1,...,n}$. Let $X=X_1$ with probability $a$ and $X=X_2$ with probability $(1-a)$.…
Jack2019
  • 1,525
0
votes
1 answer

Relationship between mutual information and entropy

I know that for two system $X$ and $Y$, we can write : $$ H(X,Y)=H(X)+H(Y)-I(X,Y)$$ Where $I$ is called the mutual information and $H$ is the shannon entropy. My question is : do we have another equation constraining $I$, or is it the only one…
StarBucK
  • 689
0
votes
0 answers

Calculating entropy of known sequence like 0,1,2,3 ...

If we apply Shannon's formula to a sequence of numbers that we know how to generate (for example, natural numbers), shouldn't entropy be 0? Mine might be a too intuitive definition of entropy. But if something is predictable, does it have entropy at…
Pierre B
  • 119
0
votes
2 answers

How to calulate entropy $H(X+S+Z,X+\alpha S)$

I have no idea how to calulate entropy $H(X+S+Z,X+\alpha S)$ where $X\sim N(0,P)$, $S\sim N(0,Q)$, $Z\sim N(0,N)$ and $\alpha$ is a constant number.Here we use $N$ to denote Gaussian distribution. Can anyone hint me?
Youninfaty
  • 45
0
votes
1 answer

Problem about cross entropy

I would like to know why rewriting $$- x * z + \log(1 + \exp(x))$$ as $$\max(x, 0) - x * z + \log(1 + \exp(-|x|))$$ can ensure stability and avoid overflow? Also, what is meant by stability? Finally, why does the rewriting avoid overflow?
PatrickL
  • 11
0
votes
1 answer

Proving $H(X,Y)\leq H(X)+H(Y)$

Knowing that $$H(X)=-\sum_{x,y} p(x,y) \log p(x)$$ and $$H(Y)=-\sum_{x,y} p(x,y) \log p(y)$$ Show that $H(X,Y)\le H(X)+H(Y)$ This is what I have solved so far, by using theory, I can simply say that $H(X,Y)=H(X)+H(Y\mid X)$ and therefore, $H(Y\mid…
Data HV
  • 1
0
votes
0 answers

Relation between Shanon entropy via relation of probabilities

We have two vectore of probabilities, $P=\left(p_1,p_2,p_3,p_4\right)$ and $Q=\left(q_1,q_2,q_3,q_4\right)$. Assume that we know only the ralation between $p_i$ and $q_i$. To be more specific we know that $p_1 >q_1, p_4>q_4, p_2
K.D.
  • 111
0
votes
1 answer

Entropy of cosine similarity etc.

First, I would like to know how to calculate entropy of cosine similarity. We have $H(X)=-\sum P(x_i)log_bP(x_i)$ for entrpy, and similarity $ = \sum \frac{A_iB_i}{\sqrt{\sum(A_i^2)}\sqrt{\sum(B_i^2)}} $, but we have two variable A,B for cosine…
cosmiccapsule
  • 51
0
votes
1 answer

Calculating entropy Through Huffman Codeword lengths.

Given the length of the codeword (i.e. the binary representation of a characters: $1, 1010, 00$, etc.) of each symbol in an alphabet, how could I calculate the bit per symbol entropy? The particular problem I'm solving has the alphabet…
NUGA
  • 21
0
votes
2 answers

Why is the conditional entropy defined as $H(Y\mid X) = \sum_{x\in X} p_X(x) H (Y\mid X = x)$

In the book "Elements of Information Theory" $H(Y\mid X)$ is defined like that and then it's shown that this is \begin{align*} H(Y\mid X) &= \sum_{x\in X} p_X(x) H (Y\mid X = x) \\ &= - \sum_{x\in X} \sum_{y \in Y} p(x,y) \log_2 p(y\mid x) \\ &=…
Stefan Falk
  • 1,217
  • 1
  • 11
  • 24
1 2 3 4
5
6