The Multi-Layer Perceptron
Stack neurons into layers, add non-linearity, and suddenly XOR is easy.
From one neuron to a network
A single perceptron can only draw a straight line. An MLP (multi-layer perceptron) feeds the input through one or more hidden layers. Each hidden neuron computes a weighted sum followed by a non-linear activation. The composition of many bent functions can approximate any continuous function — that's the universal approximation theorem.
This same shape — linear, non-linearity, linear — is exactly what transformers call the feed-forward network (FFN). When you later see "FFN" inside a transformer block, it is literally a small MLP applied to each token.
Train one — for real
Below is a genuine MLP (2 → hidden → 1) implemented in JavaScript, trained with backpropagation and gradient descent while you watch. The background shows the network's output over the whole plane — watch the boundary bend, which a single perceptron could never do. Try XOR first.
Inside the network: the forward pass
Hover over the plot above (or drag on it) and this diagram animates the forward pass for that exact input point, using the live weights of the network you just trained. Edge thickness/color = weight sign and magnitude; node brightness = activation strength.
Blue edges = positive weights, red = negative. This is the live state of the model above.
How it learns: backpropagation
Training minimizes a loss function (here, binary cross-entropy) by computing the gradient of the loss with respect to every weight using the chain rule — backpropagation — and stepping weights downhill:
Deeper stacks of these layers, plus tricks to keep gradients healthy, are what "deep learning" means. Next up: the activation functions that make the bending possible.