Self-Attention

The mechanism that lets every token look at every other token — the heart of the transformer.

The idea

An MLP processes one input on its own. But words get meaning from context: "bank" means something different in "river bank" vs "bank account". Self-attention lets each token build its representation by mixing in information from the other tokens — weighted by how relevant they are.

Every token embedding is projected into three vectors:

Q = X·W_Q (query: "what am I looking for?")
K = X·W_K (key: "what do I contain?")
V = X·W_V (value: "what do I pass along if selected?")
Attention(Q, K, V) = softmax(QKᵀ / √dk) · V

The √dk is scaling: dot products grow with dimension, and dividing keeps the softmax from saturating. In decoder models (GPT-style) a causal mask hides future tokens so the model can't cheat when predicting the next word.

Step through it

This demo computes attention for real over the sentence below (toy 8-dimensional embeddings, one head). Press ▶ Animate to watch the four stages, or drag the stage slider. Hover a token to see who it attends to; hover the heatmap to inspect individual scores. Toggle the causal mask to see the GPT-style lower-triangular pattern appear.

weighted sum of V
Query Key Value attention weight (bright = high)

The attention matrix

Row i shows how token i distributes its attention over all tokens — each row sums to 1 (that's the softmax). With the causal mask on, the upper triangle is −∞ before softmax, so it comes out exactly 0.

Hover a cell to inspect it.

Multi-head attention

Real models run many attention heads in parallel — each with its own W_Q, W_K, W_V — so different heads can specialize (one tracks syntax, another coreference, another position). Their outputs are concatenated and projected back down. GPT-2 small: 12 heads × 64 dims each. Press 🎲 above to re-roll this head's weights and see how the pattern changes — that's the difference between two heads.

Attention alone just mixes vectors; the FFN then transforms each one. Put attention + FFN + residuals + LayerNorm together and you get a transformer block — next page.