Also known as: transformer architecture, attention is all you need
The transformer is the neural-network architecture behind every modern LLM, embedding model, and reranker. Its defining feature is self-attention.
The transformer, introduced in Attention Is All You Need (Vaswani et al., 2017), is the neural-network architecture that underlies essentially every modern language model — LLMs, embedding models, rerankers, code models, vision-language models. The 2017 paper proposed it as a translation architecture; within five years it had displaced every prior approach across language tasks and most of computer vision.
A transformer is a stack of identical “blocks”. Each block does two things, in order:
Both are wrapped in residual connections and layer normalization. Stack 12 to 200 of these blocks, train on huge data, and you get GPT, Claude, BERT, T5, Llama — they all use this pattern with mostly cosmetic variations.
Three reasons it dominated:
[query, document] concatenated. Scores the joint pair.The killer is the
The fundamental cost is unchanged. What’s changed is the engineering around it: FlashAttention reorders the computation so the N x N matrix never materializes in slow memory, dropping memory cost from quadratic to linear; grouped-query attention shrinks the KV cache by sharing K/V heads across query heads; sliding-window or sparse-attention variants make most attention pairs zero by construction; and prompt-caching reuses the K/V state across calls so only the new tokens pay attention’s cost.
Stack these and a 1M-token context becomes feasible — but per-token compute and memory are still roughly proportional to the active attention pairs you keep, which is why long-context inference remains 10-100x more expensive per generated token than short-context inference. The wall is bent, not broken.
RNNs process tokens sequentially — token N can only see tokens 1..N-1 through a single hidden-state bottleneck. Transformers let every token attend directly to every other, which both captures long-range dependencies better and parallelizes across the sequence at training time. Modern hardware loves the second part.
Encoders (BERT, embedding models, [bi-encoders](/concepts/bi-encoder/)) read a sequence and produce representations — they look both ways. Decoders (GPT, Claude, most LLMs) generate token-by-token left-to-right. Encoder-decoders (T5, original transformer) read a source then generate a target — common in translation. The architecture is the same primitives, different masking.
Self-attention is O(n²) in sequence length — both compute and memory grow quadratically. Various tricks (sliding-window attention, sparse attention, RoPE, FlashAttention) reduce the constant or shape of this scaling, but it remains the dominant constraint on practical context window length.
