LLM Inference
Prefill, decode, and the KV cache — how a trained transformer actually serves you tokens.
Two very different phases
When you send a prompt to an LLM, generation happens in two phases with completely different performance characteristics:
Prefill processes your entire prompt in one parallel pass. Every token's keys and values are computed simultaneously — a huge matrix multiply that saturates the GPU's compute units. This is why prefill is compute-bound, and why "time to first token" grows with prompt length.
Decode then generates one token at a time. Each new token attends to all previous tokens — but instead of recomputing their K and V, it reads them from the KV cache. Each step must still stream all the model weights from memory to produce a single token, so decode is memory-bandwidth-bound: the GPU's compute sits mostly idle waiting on memory.
Watch it generate
Press ▶ Run. During prefill, watch all prompt columns of the KV cache fill at once. Then decode begins: each step, attention arrows read every cached column, one new token appears, and exactly one new column is appended to the cache. The cache grid shows one row per layer — this is real memory in a real deployment (for Llama-3-70B, roughly 320 KB per token).
Toggle "use KV cache" off and re-run: total work explodes from linear to quadratic — that's why every serving stack caches K and V.
Picking the next token: sampling
Each decode step ends with a probability distribution over the whole vocabulary. How you pick from it defines the model's personality. Drag the temperature slider: low T sharpens the distribution toward greedy, deterministic output; high T flattens it toward creative chaos. Top-k cuts the long tail before sampling.
Why this matters at the frontier
Serving economics are dominated by these two phases. The KV cache for one long-context request can be gigabytes — often more than the activations — so systems like vLLM invented PagedAttention (virtual-memory-style paging for cache blocks). Grouped-query attention shrinks the cache by sharing K/V across query heads (Llama-3: 8 KV heads serve 64 query heads). Speculative decoding attacks the sequential bottleneck: a small draft model proposes several tokens and the big model verifies them in a single parallel pass — prefill-style compute efficiency applied to decode. Batching many users' decode steps together is what makes serving profitable at all.
You've reached the frontier. For the full vocabulary tour, head to the glossary.