8

In this paper Boltzmann Machines, the first paragraph of the second section 2. Boltzmann Machines argues that,

we could never train such a network with three units to visit the states (0,0,0), (0,1,1), (1,0,1) and (1,1,0) with some probabilities, but not the other four possible states. This is because the first and second order statistics (the mean of each component and correlations between components) are the same for these four states as the remaining four and so the network can not discriminate them.

As I learned from here, order logic is just to order the attributes of the samples and take the i-th element of it.

I am wondering why the first and second order statistics of the two sets are the same.

Thank you!

Jiang Xiang
  • 257
  • 1
  • 2
  • 12

2 Answers2

13

I know that this is an old question with an accepted answer. But in the hope of helping someone finding this question now or in the future I'm answering.

I believe the paper was referring to "low-order statistics" and not "order statistics". Low-order statistics are functions or quantitative measures which use the zeroth, first and second power of a sample.

For a probability density, the first-order statistic (first moment) is the mean, the second-order statistic (second moment) is the variance.

cyuut
  • 131
  • 1
  • 4
  • I agree, they even mention it as such. I came here to clear the concept up that I have seen in another paper "Covariance Pooling for Facial Recognition" which most certainly refers to Low-order statistics and not $i$-th order as they are defined in OP's link – daniels_pa Dec 27 '18 at 16:10
5

On the surface, it seems like the point they're trying to make has more to do with identifiability than order statistics. Perhaps, in their language, "order statistics" refers to moments (in statistics) or powers of a sample. This is evidenced by their statement in parentheses about the mean and covariance.

If they are actually referring to moments, and a finite set of moments drive the states in question (e.g. as a normal distribution is fully determined by its mean and [co]variance), then they might simply be describing the model's inherent symmetry and/or limitations by way of its low-order moments.

Also, it sounds like they're voicing concerns similar to those that arise when considering the identifiability of mixture models.

brandonwillard
  • 66