Pruning

Pruning is a model compression technique in machine learning and deep learning designed to reduce the size and computational complexity of Neural Networks by removing redundant or unimportant parameters (such as weights, neurons, or filters). The primary goal is to enable efficient deployment—particularly on resource-constrained hardware like edge devices and mobile systems—without significantly compromising model accuracy.

Key Features

Pruning methods are generally categorized by what is removed, when it is removed, and how parameter importance is determined:

1. By Structure (What is Pruned)

• Unstructured Pruning (Fine-grained): Removes individual weights based on an importance criterion (commonly magnitude). While it allows for high compression rates, it often results in sparse matrices that require specialized hardware or libraries to achieve real-world speed improvements.
• Structured Pruning (Coarse-grained): Removes entire structural components, such as neurons, channels, or convolutional filters. Because it preserves the original network topology (producing a smaller but dense model), it is generally more hardware-friendly and easily accelerated using standard deep learning frameworks.

2. By Schedule (When Pruning Occurs)

• One-shot Pruning: Pruning occurs in a single step (often after training). It is generally efficient at lower pruning ratios where only a small portion of the model is removed.
• Iterative Pruning: Pruning and fine-tuning are performed in multiple cycles. This is typically more effective for aggressive, high-ratio pruning, as it allows the model to recover from the loss of parameters incrementally.

3. Importance Criteria (How Parameters are Chosen)

• Magnitude-based: The most common approach, which ranks weights by their absolute value (e.g., $L_1$ or $L_2$ norm) and removes those below a certain threshold.
• Sensitivity/Gradient-based: Uses gradient information or sensitivity analysis to determine which parameters contribute least to the loss function.

---

Related Topics

Pruning is a core component of the broader field of Model Compression and optimization, which often combines multiple techniques:

• Quantization: Reducing the precision of weights (e.g., from FP32 to INT8). Often applied after pruning.
• Knowledge Distillation: Training a smaller “student” model to mimic a larger “teacher” model to recover lost accuracy.
• Lottery Ticket Hypothesis: The theory that dense, randomly initialized networks contain sparse subnetworks (“winning tickets”) that can be trained to reach comparable performance.
• Low-rank Factorization: Decomposing large weight matrices into smaller, lower-rank matrices.