Transformer Architecture
The transformer architecture is a self-attention based sequence-to-sequence model architecture introduced in 2017.
Key Features
- Uses self-attention instead of recurrence
- Can process entire sequences in parallel
- Introduced in “Attention Is All You Need” (2017)
Self-Attention
Self-attention computes query, key, value matrices for each token. Attention scores are calculated based on dot product similarity between queries and keys.
Related Concepts
- karpathy-llm-wiki - Karpathy’s wiki pattern for persistent knowledge bases
- attention-is-all-you-need - The 2017 paper introducing transformers