The Transformer Block
"Attention Is All You Need" (2017) — the architecture behind every modern LLM.
One block, repeated N times
A GPT-style (decoder-only) transformer is astonishingly uniform: an embedding layer, then the same block stacked N times (GPT-2 small: 12; Llama-3-70B: 80), then a final projection to vocabulary logits. Each block does just two things — attention (tokens talk to each other) and an FFN (each token thinks by itself) — wrapped in residual connections and normalization.
Explore the architecture
Click any component in the diagram to read what it does. Press ▶ Flow to watch a token's vector travel through the block — note how the residual stream (the vertical line) carries it past every sublayer, with attention and FFN only adding refinements to it.
👆 Click a component
Every box in the diagram is clickable. Start at the bottom (tokens enter there) and work upward — that's the direction data flows.
The thick vertical line is the residual stream — think of it as a shared whiteboard that every sublayer reads from and writes small updates onto.
Why this design won
Parallelism: unlike RNNs, every token is processed simultaneously — perfect for GPUs. Long-range: any token can attend to any other in one hop, no matter the distance. Scale: the recipe barely changes from 100M to 1T+ parameters; you mostly increase d_model, heads, and layers. The scaling laws that emerged from this uniformity are what launched the LLM era.
Modern variants tweak details — Pre-LN (norm before the sublayer, as drawn here) for stable training, RMSNorm instead of LayerNorm, RoPE instead of learned positions, mixture-of-experts FFNs — but the block you see above is still recognizably the same machine. Next: how this machine actually generates text.