Self-Attention

Self-attention (also known as scaled dot-product attention) is an attention mechanism that relates different positions of a single sequence in order to compute a representation of the sequence. It serves as the foundational building block of the Transformer architecture, which has revolutionized the field of Deep Learning and Natural Language Processing.

Key Features

1. Long-Range Dependency Resolution: Unlike Recurrent Neural Networks (RNNs) that process sequences sequentially, self-attention computes relationships between all pairs of tokens in a sequence directly. The path length between any two tokens is $O(1)$, which significantly improves the model’s ability to learn long-range context without suffering from vanishing or exploding gradients.
2. Parallelizability: Since the computation of attention weights for all tokens does not depend on prior tokens, the operations can be fully parallelized across hardware accelerators like GPUs and TPUs.
3. Dynamic Context-Aware Representations: The representation generated for each token is dynamic and context-dependent. For instance, the pronoun “it” will have a representation highly weighted toward the noun it refers to in that specific sentence.
4. Scale Invariance (via Scaling): By scaling the dot product by the square root of the key dimension ($\sqrt{d_k}$), it prevents dot products from growing excessively large, ensuring stable gradient flow during training.

Inner Working & QKV Mechanism

Self-attention operates by projecting the input embeddings into three distinct vectors using learned projection matrices:

• Query ($Q$): The representation of the current token looking for relevant context.
• Key ($K$): The representation of all tokens acting as candidates to match against the query.
• Value ($V$): The actual content representation of each token, which is aggregated to form the output.

Mathematical Formulation

The mathematical equation for scaled dot-product attention is defined as:

$\text{Attention}(Q,K,V)=\text{softmax}\left(\frac{Q{K}^T}{\sqrt{d_k}}\right)V$

Where:

• $Q,K,V$ are the matrices of queries, keys, and values.
• $d_k$ is the dimension of the key vectors.
• $\text{softmax}$ is applied row-wise to obtain a probability distribution over the sequence positions.

Related Topics

• Transformer: The modern neural network architecture built entirely on self-attention mechanisms.
• Multi-Head Attention: An extension where self-attention is computed multiple times in parallel with different learned linear projections, allowing the model to jointly attend to information from different representation subspaces at different positions.
• Cross-Attention: A variant where queries come from one sequence (e.g., decoder targets) and keys/values come from another (e.g., encoder outputs), common in sequence-to-sequence tasks like machine translation.
• Natural Language Processing: The domain where self-attention first achieved widespread success, powering models like BERT and GPT.