Mathematics Stack Exchange
Mathematics Stack Exchange

Questions tagged [entropy]

This tag is for questions about mathematical entropy. If you have a question about thermodynamical entropy, visit Physics Stack Exchange or Chemistry Stack Exchange instead.

This tag is for questions about mathematical entropy, not to be confused with thermodynamical entropy which goes in Physics or Chemistry Stack Exchange.

1617 questions
1
vote
0 answers

Entropy calculation of Fibonacci distribution

For any positive integer $N$, consider the Fibonacci sequence $F_n$ of length $N$. Using $F_n$ we can define a Fibonacci discrete probability distribution as follows: $$p_N(n)=\frac{F_n}{\sum_{k=1}^N F_k}\ \ \forall…
Anthony
  • 53
0
votes
1 answer

Meaning of this term:$H(X \oplus\hat{X}|\hat{X} )$

Here, $H$ means the entropy function. I understand that the symbol $\oplus$ means modulo $2$ addition. But I don't understand the significance of the entire expression.
sprajagopal
  • 622
  • 1
  • 5
  • 20
0
votes
0 answers

Can we calculate entropy for nonstationary random variables?

Let's assume that $X$ is a discrete random variable, which can take any value from the set $\{x_0,\dots,x_n\}$ with the probability mass function $P(X)$. We can calculate the entropy of $X$ as follows, $\text{Ent}(X) = -\sum\limits_{i=1}^n {P(x_i)…
Rasa
  • 357
0
votes
2 answers

Interpreting Password Entropy Calculation – Property of Character Entropy

I was reading this explanation on how to calculate the entropy of a password. The article is great and it explains it very succinctly that even I understood it. According to the site, if you have a password that has only lower-case characters, you…
Lex
  • 103
0
votes
0 answers

Don't understand a line in a paper by Horibe

This question pertains to "A Entropy view of Fibonacci Trees" (1982) by Yasuichi Horibe, Zbl 0491.94009. Horibe defines a binary tree where a node has a left branch with probability x and a right branch with probability $1-x$. The left branch has…
William Butler
0
votes
1 answer

why the entropy of a data set is not defined as H(p)=p(1-p)?

I'm learning some ML algorithm online, which talks about the use of entropy as a measurement of impurity of the dataset. Suppose the dataset contains two types of objects: objects with positive label and objects with negative label. The entropy of…
ExcitedSnail
  • 503
0
votes
0 answers

How to calculate the entropy of bigrams of a language H(X|X)

I have a table with a check for each bigram (Sum of all probabilities $= 1$, and also approximately converges with similar tables on the Internet) https://prnt.sc/mmorTG2HcZ3D I need to calculate the conditional entropy, for this I use the…
Forsaken_Oracle
  • 1
0
votes
1 answer

Is a entropy the higher the better or the lower the better?

I'm new to this field. To my knowledge, entropy is defined so that the higher the value is, the more Information or uncertainty it contains. But I also noticed that the uniform distribution maximized the entropy, which seems not to be so uncertain.…
toki
  • 95
0
votes
1 answer

A question about the convexity of entropy

Nowadays I refer to some references about entropy.They all say "The entropy function $H(X)$ is a concave function".The definition is as follows: Let $X$ be a continuous random variable with probability density function (pdf) $f (x)$ (in short $X ∼ f…
solver
  • 145
  • 6
0
votes
1 answer

How much entropy exists in knowing the specific sorting of a list of $n$ many items with repetition in the list?

If I have a bag of $b$ many balls, each numbered from $1, 2, \ldots, b$, and I uniformly-randomly pick one ball. Then I ask you "how much information would you gain should I tell you the ball which I have selected?". Your answer will be…
caveman
  • 393
  • 1
  • 13
0
votes
1 answer

maximum value of $\sum_i a_i \log b_i $ when $\sum a_i = \sum b_i = 1$

$J=\sum_i a_i \log b_i $ and $\sum_i a_i = 1, \sum_i b_i = 1 $ when $a_i = b_i $, $J$ has a maximum value.(Is this true?) How to prove this?
plhn
  • 631
0
votes
1 answer

Help with a solution about entropy!

I'm triying to solve this question question, but I don't understand why $$ -\log_2 (2\pi n pq)/(2n) $$ transforms into $$O(\ln n/ n)$$ can you help me please :( !!
Faby V
  • 1
0
votes
1 answer

range of variation for a conditional entropy

Elements of alphabets $X$ and $Y$ are statistically related. It is known that $H(X)=4$ bits and $H(Y) =11$ bits. What are a range of variation for a conditional entropy $H(Y|X)$ and $H(X|Y)$ changes from min to max?
Sultan Akhmetbek
  • 1
0
votes
0 answers

$H[X_1, .., X_{n-1}|X_n] = H[X_n , ..., X_2|X_1]$ for a stationary source

I posted this question before but i want to give more context. I have the following theorem For a stationary source $X_1,..,X_n$, the term $H(X_n|X_{n−1}, . . . ,X_1)$ is nonincreasing in n and has a limit $\lim_{n \rightarrow \infty} H[X_n|X_{n-1},…
Nic2431
  • 13
  • 3
0
votes
1 answer

Understanding mutual information

Given random variables $\vec{x}, \vec{y} \in \mathbb{R}^n$, is it true that $I(\vec{x}: \vec{y}) \geq \sum_i I(x_i, y_i)$ My interpretation is that that collectively several variables should be able to predict another set of variables at least as…
Aleksejs Fomins
  • 2,240
1 2 3
4
5 6