Transformer

The Transformer is a state-of-the-art Deep Learning architecture introduced in June 2017 by Google researchers in the seminal paper “Attention Is All You Need”. It revolutionized Natural Language Processing (NLP) and computer vision by replacing sequential architectures (such as Recurrent Neural Networks (RNNs) and LSTMs) with a fully attention-based mechanism.

Key Features

• Self-Attention Mechanism: The core innovation of the Transformer. Instead of processing inputs sequentially, the self-attention mechanism computes a representation of each token by attending to all other tokens in the sequence simultaneously, regardless of their distance.
• Parallelization: By avoiding recurrence, the Transformer allows sequences to be processed in parallel during training. This dramatically accelerates training times and scales efficiently on modern hardware (like GPUs and TPUs).
• Positional Encoding: Because attention treats inputs as a bag of words without inherent order, positional encodings (sine and cosine wave vectors) are added to the input embeddings to convey sequence order.
• Multi-Head Attention: Allows the model to jointly attend to information from different representation subspaces at different positions, enhancing the richness of learned features.

Core Components

The original Transformer follows an encoder-decoder structure:

1. Input Embeddings & Positional Encodings: Maps tokens to continuous vectors and adds positional information.
2. Encoder: Composed of a stack of identical layers, each containing a multi-head self-attention layer followed by a position-wise feed-forward network.
3. Decoder: Composed of a stack of identical layers, adding a masked multi-head self-attention layer (to prevent looking at future tokens) and an encoder-decoder attention layer (to attend to the encoder’s output).
4. Layer Normalization and Residual Connections: Surrounds each sub-layer to stabilize gradient flow and training dynamics in deep networks.

Related Topics & Applications

• Large Language Models (LLMs): Modern models like GPT-4, LLaMA, and Claude are based on decoder-only Transformer architectures.
• Bidirectional Representations (e.g., BERT): An encoder-only variant designed for pre-training deep bidirectional representations from unlabeled text.
• Vision Transformers (ViT): Adapts the Transformer architecture for computer vision tasks by treating image patches as tokens.
• Sequence-to-Sequence (Seq2Seq): Models like T5 (Text-to-Text Transfer Transformer) retain the original encoder-decoder structure for tasks like translation and summarization.