Variational Inference

Variational Inference (VI) is a powerful statistical technique in machine learning and Bayesian statistics used to approximate complex, intractable probability distributions. Instead of relying on sampling-based methods like markov-chain-monte-carlo (MCMC), Variational Inference frames the approximation task as a deterministic optimization problem.

Key Features

• Inference as Optimization: VI translates the problem of calculating complex posterior probability distributions into an optimization task. This allows the application of standard optimization algorithms, making the process highly deterministic and often faster than sampling-based techniques.
• Evidence Lower Bound (ELBO): Because the true posterior distribution is typically intractable to compute directly, VI defines and maximizes a surrogate objective called the Evidence Lower Bound (ELBO). Maximizing the ELBO is mathematically equivalent to minimizing the kullback-leibler-divergence (KL divergence) between the approximating distribution and the true posterior.
• Tractable Family Approximation: The method selects a family of tractable probability distributions (e.g., Gaussian distributions under the mean-field approximation where latent variables are assumed to be mutually independent) and optimization parameters are tuned to find the best fit within this family.
• Scalability and Trade-offs: Compared to MCMC, VI scales significantly better to large datasets and high-dimensional models (especially when combined with stochastic gradient methods, known as Stochastic Variational Inference). However, it introduces a systematic bias, frequently underestimating the variance of the true posterior.

Related Topics

• bayesian-inference: VI is a core method for approximate Bayesian inference in models where exact calculation of the posterior is computationally prohibitive.
• variational-autoencoder (VAEs): Deep generative models that integrate neural networks with variational inference by optimizing the ELBO to learn low-dimensional latent representations.
• markov-chain-monte-carlo (MCMC): The primary alternative to VI; a class of algorithms for sampling from a probability distribution.
• mean-field-theory: A simplifying assumption in VI where latent variables are assumed to be independent in the approximating distribution, which simplifies calculations but can limit the model’s capacity to represent correlations.
• stochastic-gradient-descent: Frequently used in Stochastic Variational Inference (SVI) to enable scaling to massive datasets.