Kullback-Leibler Divergence

Kullback-Leibler (KL) divergence, also known as relative-entropy, is a fundamental measure in information theory and mathematical statistics that quantifies how much one probability distribution differs from a reference probability distribution.

In machine learning and statistics, it is typically used to measure the difference between a “true” data distribution $P$ and an approximating model distribution $Q$.

---

1. Definition & Mathematical Formulation

For two probability distributions $P$ and $Q$ defined over the same probability space, the KL divergence from $Q$ to $P$ is defined as:

Discrete Distributions

$D_{\text{KL}}(P\parallel Q)={\sum }_{x\in \mathcal{X}}P(x)\log \left(\frac{P(x)}{Q(x)}\right)$

Continuous Distributions

$D_{\text{KL}}(P\parallel Q)={\int }_{-\infty }^{\infty }p(x)\log \left(\frac{p(x)}{q(x)}\right)dx$

Note: In these formulations, $P$ represents the true distribution or target data distribution, while $Q$ represents the approximation, model, or prior distribution.

---

2. Key Features & Properties

• Non-Negativity (Gibbs’ Inequality): $D_{\text{KL}}(P\parallel Q)\ge 0$. The divergence is strictly non-negative, and it equals $0$ if and only if $P=Q$ almost everywhere.
• Asymmetry (Non-Symmetric): $D_{\text{KL}}(P\parallel Q)\ne D_{\text{KL}}(Q\parallel P)$. Because the order of distributions matters, KL divergence is not a true distance metric.
• No Triangle Inequality: Unlike true mathematical metrics, KL divergence does not satisfy the triangle inequality.
• Additive for Independent Distributions: If $P_1,P_2$ and $Q_1,Q_2$ are independent joint distributions, then $D_{\text{KL}}(P_1{P}_2\parallel Q_1{Q}_2)=D_{\text{KL}}(P_1\parallel Q_1)+D_{\text{KL}}(P_2\parallel Q_2)$.

---

3. Related Topics & Information Theory Connections

• cross-entropy: The cross-entropy $H(P,Q)$ between two distributions is directly related to the Shannon entropy $H(P)$ and KL divergence: $H(P,Q)=H(P)+D_{\text{KL}}(P\parallel Q)$ In many deep learning contexts (such as classification tasks), minimizing the cross-entropy loss is equivalent to minimizing the KL divergence since the true data distribution’s entropy $H(P)$ is constant.
• Mutual Information: The mutual information between two random variables $X$ and $Y$ can be expressed as the KL divergence between their joint distribution and the product of their marginal distributions: $I(X;Y)=D_{\text{KL}}(P(X,Y)\parallel P(X)P(Y))$
• Jensen-Shannon Divergence: A symmetric and bounded alternative to KL divergence, defined as the average KL divergence of $P$ and $Q$ to their average distribution.

---

4. Key Applications in AI/ML

• variational-autoencoders (VAEs): In variational inference, VAEs utilize KL divergence as a regularization term in the Evidence Lower Bound (ELBO) loss. It constrains the learned latent representation distribution $Q(z|x)$ to be close to a known prior distribution $P(z)$ (typically a standard Gaussian).
• reinforcement-learning: Policy optimization algorithms such as Trust Region Policy Optimization (TRPO) and Proximal Policy Optimization (PPO) employ KL divergence constraints to ensure policy updates do not deviate too far from the previous, stable policy.
• Data Drift and Model Monitoring: In production environments, KL divergence is used to monitor input distributions and detect covariate/concept drift between training baseline distributions and real-world inference data.