Low-Rank Adaptation (LoRA)

Low-Rank Adaptation (LoRA)

Low-Rank Adaptation (LoRA) is an innovative parameter-efficient fine-tuning (PEFT) technique introduced by Edward Hu et al. in 2021. It addresses the computational and storage bottlenecks of adapting massive pre-trained models to specific downstream tasks.

Core Concept & Mathematical Formulation

Traditional Fine-Tuning updates the entire weight matrix $W_0\in {\mathbb{R}}^{d\times k}$ of a neural network layer by calculating a gradient step $\Delta W$. In contrast, LoRA assumes that the weight updates during adaptation have a low “intrinsic dimension” or rank.

LoRA parameterizes the update $\Delta W$ using two low-rank decomposition matrices $A\in {\mathbb{R}}^{d\times r}$ and $B\in {\mathbb{R}}^{r\times k}$, where the rank $r\ll \min (d,k)$: $W=W_0+\Delta W=W_0+\frac{\alpha }{r}BA$ During training, the pre-trained weight matrix $W_0$ is frozen (receives no gradient updates), while only $A$ and $B$ are trainable. $A$ is typically initialized using a random Gaussian distribution, and $B$ is initialized to zero, ensuring $\Delta W=0$ at the start of training. $\alpha$ is a constant scaling factor.

Key Features

• Reduced Trainable Parameters: By adapting only the low-rank matrices, LoRA can reduce the number of trainable parameters by up to 10,000x for large models like GPT-3 175B.
• Zero Inference Latency: Since the learned updates $\Delta W=BA$ are of the same dimensions as the original weight matrix, they can be merged directly back into the base model ($W=W_0+BA$) before deployment. This incurs zero latency overhead at inference time.
• VRAM and Storage Savings: Freezing the base model drastically reduces optimizer states (such as those in Adam), dropping GPU memory usage by up to 3x. Task-specific adapters are only a few megabytes in size, facilitating cheap storage.
• Modular Adaptation: The same frozen base model can be deployed alongside multiple task-specific LoRA adapters. Adapters can be dynamically swapped in and out of GPU memory depending on the input request.

Prominent Extensions

• QLoRA (Quantized LoRA): Quantizes the base model to 4-bit precision (NormalFloat4) and performs backpropagation through it into 16-bit LoRA adapters. This enables fine-tuning 65B+ parameter Large Language Models on single consumer GPUs.
• DoRA (Weight-Decomposed Low-Rank Adaptation): Decomposes weights into magnitude and directional components and applies LoRA updates specifically to the directional component, aligning learning dynamics closely with full fine-tuning.
• AdaLoRA (Adaptive LoRA): Dynamically allocates parameter budgets (ranks) across different layers based on importance, pruning low-rank matrices with negligible singular values.

Related Topics

• Parameter-Efficient Fine-Tuning (PEFT)
• Transfer Learning
• Singular Value Decomposition (SVD)
• Quantization (machine learning)