Variational Autoencoder

A Variational Autoencoder (VAE) is a class of generative models first proposed by Diederik Kingma and Max Welling in 2013. It integrates neural networks with probabilistic graphical models, combining the compression power of deep learning with the rigorous framework of Bayesian inference. Unlike traditional Autoencoder models that map inputs to discrete, deterministic points in a bottleneck layer, VAEs learn the parameters of a probability distribution representing the data, mapping inputs to a continuous Latent Space.

Key Features

• Probabilistic Latent Space: Instead of encoding inputs as static vectors, a VAE outputs statistical parameters: a mean vector ($\mu$) and a variance/log-variance vector (${\sigma }^2$). This maps inputs to a probability distribution (typically a multivariate Gaussian), ensuring that the latent space remains smooth and continuous.
• Generative Capability: Because the latent space is continuous and follows a standard distribution, new, realistic data points can be generated by sampling vectors directly from the latent distribution and feeding them to the decoder.
• The Reparameterization Trick: Stochastic sampling ($z\sim q_{\varphi }(z|x)$) is non-differentiable, preventing backpropagation. The reparameterization trick solves this by shifting the randomness to an auxiliary noise variable $ϵ\sim \mathcal{N}(0,I)$, formulating the latent representation as $z=\mu +\sigma \odot ϵ$, which allows gradients to flow back to the encoder.
• Regularized Objective Function: The model is optimized using the Evidence Lower Bound (ELBO), balancing reconstruction accuracy with the structural shape of the latent space.

Mathematical Formulation

The VAE loss function is defined as: ${\mathcal{L}}_{\text{VAE}}(\theta ,\varphi ;x)=-{\mathbb{E}}_{q_{\varphi }(z|x)}[\log p_{\theta }(x|z)]+D_{KL}(q_{\varphi }(z|x)\parallel p(z))$

Where:

1. Reconstruction Loss ($-{\mathbb{E}}_{q_{\varphi }(z|x)}[\log p_{\theta }(x|z)]$): Measures the decoder’s ability to reconstruct the input $x$ from the latent representation $z$.
2. Kullback-Leibler (KL) Divergence ($D_{KL}(q_{\varphi }(z|x)\parallel p(z))$): Measures how much the learned distribution $q_{\varphi }(z|x)$ deviates from the prior distribution $p(z)$, which is typically a standard isotropic Gaussian ($\mathcal{N}(0,I)$). This acts as a regularizer to prevent the model from assigning infinite variances or clustering samples too far apart.

Applications

• Generative Modeling & Synthesis: Creating new images, music, or speech samples by sampling from the latent space.
• Anomaly Detection: Detecting out-of-distribution samples by monitoring reconstruction errors or likelihoods.
• Representation Learning: Acquiring low-dimensional, disentangled representations of high-dimensional data for downstream tasks (e.g. classification).
• Data Denoising: Filtering noisy inputs to recover clean signals.

Related Topics

• Autoencoder: The deterministic precursor to the VAE.
• Generative Adversarial Network (GAN): An alternative class of generative models that uses a game-theoretic approach to generate realistic data.
• Diffusion Model: Modern state-of-the-art generative models that iteratively denoise random signals.
• Unsupervised Learning: The broader paradigm of machine learning to which VAEs belong.
• Latent Variable Model: A statistical model that relates a set of observable variables to a set of latent variables.