1

As new to variational autoencoder, there are some simple details perplex me. The basic idea of VAE is to use an encoder to map some unknown distribution (e.g. mnist images) to a specific distribution like Gaussian, and then decode this latent distribution back to the original distribution. In Kingma's paper, I feel confused about these simple facts:

  1. The paper assume p(x|z) to be a Gaussian distribution. But in my opinion, p(x|z) is what the decoder does to map the latent variable z to the original distribution(e.g. mnist images). Thus, p(x|z) actually represents the original distribution. Why can we just simply assume it to be a Gaussian distribution?
  2. The cost function of VAE is composed of the reconstruction error and regularization error. But I don't know what's the effect of regularization error and what if we just use the reconstruction error?
  3. The encoder of VAE outputs the mean u and s.t.d for the Gaussian distribution and then we sample z from it. But why don't we just let the encoder output z in Gaussian just as what we do in GAN(In GAN, we use a generator to directly output variable in the distribution we want)?
Jermmy
  • 13
  • Variational autoencoder means an autoencoder where the input/output are interpreted as probability distributions and the training algorithm is modified accordingly, thus aiming at minimizing some probabilistic criterion like KL divergence ? – reuns Nov 23 '17 at 01:55
  • Yeah, but I wonder why we can assume p(x|z) to be Gaussian. I think p(x|z) should output x, which is the original distribution, not Gaussian. – Jermmy Nov 23 '17 at 05:24
  • You can approximate any distribution by some Gaussian mixture, so why not assume $p(x|z) \sim \mathcal{N}(\mu_z, \Sigma_z)$ ? In which case the parameters to be learnt are the distribution for $z$ and the function $z \mapsto (\mu_z, \Sigma_z)$, ie. the distribution for $(\mu, \Sigma)$. The distribution for $x$ is $p(z) p(x|z)$. – reuns Nov 23 '17 at 05:26
  • Do you mean the decoder is somewhat like a GMM, as we will sample a z first, then input it to the decoder and output a $\mu$ and $\Sigma$ for a Gaussian, and finally sample a x from this Gaussian? – Jermmy Nov 23 '17 at 06:35

1 Answers1

2

Below are my answers to your questions. An excellent explanation of variational autoencoder can be found at in this blog.

  1. Variational autoencoder assumes the marginal likelihood of data as $$p(x) = \int dz p(x|z)p(z)$$ i.e. a continuous mixture of Gaussian. This mixture is a universal approximator for any distributions.

  2. The effect of regularization error is to make the posterior distribution of latent variable $p(z|x)$ look like a unit Gaussian. The reason we need a regularization error (aside from the fact that it comes naturally from the derivation) is we want to limit the range of latent variables so that similar data lead to similar latent variables. For example, suppose we have many different images of number "9", without the regularization error nothing can prevent the network from mapping these images to completely different value far away in the latent space, while the regularization error will make it more favorable to map these images to values close in the latent space. So you can see its similarity with other regularization techniques in neural network in here.

  3. The random sample for GAN is drawn from the prior distribution of latent variable $p(z)$, and the random sample for variational autoencoder is drawn from the posterior distribution of latent variable $p(z|x)$. If you simply want to generate an image $x$, then you do not need $p(z|x)$. You can just draw from $p(z)$ as you did in GAN and pass $z$ through the decoder network.

DiveIntoML
  • 165
  • Your second explanation is what I desired. I just come out with another question: if the distribution of original data is very different from Gaussian, will VAE fail to generate sample? – Jermmy Jan 31 '18 at 08:42
  • @Jermmy In theory VAE should still work if the distribution is different from Gaussian. If you look at the derivation of VAE, you can see that VAE wants p(x|z) to be Gaussian, and p(x) does not have to be Gaussian. One example is p(x) a mixture of two Gaussian far away from each other, and p(x) is quite different from Gaussian. The latent variable z is simply 0 or 1 indicating which Gaussian it belongs to, and p(x|z) is a Gaussian centered at corresponding center. – DiveIntoML Jan 31 '18 at 16:09