Model Quantization in Deep Learning

Model quantization is an essential model compression technique in modern deep learning that reduces the numerical precision of a model’s weights and activations (typically from 32-bit floating-point, FP32, to lower-precision formats like 8-bit integer, INT8, or 4-bit integer, INT4). This process optimizes neural networks for efficient deployment on resource-constrained hardware.

---

Key Features & Benefits

• Reduced Memory Footprint: Decreases model storage and RAM usage by up to 75% for INT8 and 87.5% for INT4/NF4, enabling deployment of large models on consumer edge devices.
• Inference Acceleration: Low-precision integer arithmetic runs faster on hardware with dedicated acceleration (e.g., Tensor Cores or DSPs) than FP32 floating-point computations.
• Lower Power Consumption: Integer operations require significantly less energy than floating-point math, critical for battery-powered mobile and IoT devices.
• Reduced Memory Bandwidth Pressure: Loading smaller weights from memory into cache requires less bandwidth, mitigating the primary bottleneck of modern large language models.

---

Mathematical Formulation

Linear quantization maps a continuous real-valued range $[r_{\text{min}},r_{\text{max}}]$ to a discrete integer range $[q_{\text{min}},q_{\text{max}}]$ using a scale factor $S$ and an integer zero-point $Z$:

1. The Core Transformation

• Quantization: $q=\text{clip}\left(\text{round}\left(\frac{r}{S}\right)+Z,q_{\text{min}},q_{\text{max}}\right)$
• Dequantization: $r\approx S\cdot (q-Z)$

2. Symmetric vs. Asymmetric Quantization

Feature	Symmetric Quantization	Asymmetric (Affine) Quantization
Zero-Point ($Z$)	Fixed at 0 ($Z=0$)	Calculated offset ($Z\ne 0$)
Mapping	Centered around zero ($[-r_{\text{max}},r_{\text{max}}]$)	Maps exact range ($[r_{\text{min}},r_{\text{max}}]$)
Scale ($S$) Formula	$S = \frac{\max(	r
Common Use Case	Weight quantization	Skewed activations (e.g., post-ReLU)

---

Primary Approaches

1. Post-Training Quantization (PTQ)

PTQ quantizes a pre-trained FP32 model without further training:

• Dynamic PTQ: Scales are calculated dynamically for activations at runtime. Extremely fast but adds overhead.
• Static PTQ: Scales are pre-calculated during a calibration phase using a small representative dataset. This eliminates runtime overhead.

2. Quantization-Aware Training (QAT)

QAT models the quantization noise during training:

• Uses fake quantization nodes to simulate rounding and clipping on the forward pass.
• Employs a Straight-Through Estimator (STE) to bypass the non-differentiable rounding function on the backward pass, allowing weights to adapt to low precision.

---

Modern LLM Quantization Techniques

With the rise of large-scale models, specialized quantization algorithms have emerged:

• GPTQ (Generalized Post-Training Quantization): A layer-wise PTQ method for GPUs that calculates the inverse Hessian matrix to compensate for quantization errors in weights.
• AWQ (Activation-aware Weight Quantization): Preserves the most salient weights (typically ~1% that correspond to large activation channels) in higher precision, resulting in better accuracy than GPTQ at ultra-low bitrates.
• GGUF (formerly GGML): A unified file format optimized for CPU-GPU offloading, standard for local LLM execution.
• QLoRA (Quantized Low-Rank Adaptation): A low-rank adaptation fine-tuning method that freezes a 4-bit base model using a specialized NormalFloat 4 (NF4) data type (which optimally fits normally distributed weights) and trains low-rank adapters.

---

Related Topics

• model compression: The overarching domain of reducing model size.
• knowledge distillation: Transferring knowledge from a large teacher model to a smaller quantized/compact student model.
• pruning: Removing redundant weight parameters or connections.
• large language models: The primary application class driving advances in modern quantization formats.