Activation Functions

Why networks need them

Composing linear functions gives you… another linear function. Without a non-linearity between layers, a 100-layer network collapses into a single matrix multiply — no better than a perceptron. The activation function, applied elementwise after each linear layer, is what lets networks bend and warp space.

Explore each function

Move your pointer across the plot to probe any input x: the readout shows f(x) and the derivative f′(x) — the gradient that flows backward during backprop. The shaded band under the curve is the derivative plotted as a filled area: where it flattens to zero, the gradient dies.

Which one when?

Sigmoid squashes to (0,1) — good for output probabilities, bad for hidden layers because its gradient vanishes away from zero. tanh is a zero-centered sigmoid — the classic pre-2012 choice. ReLU (max(0, x)) made deep networks practical: cheap, non-saturating for x>0, but neurons can "die" if they land in the flat negative region. Leaky ReLU patches that with a small negative slope. GELU and SiLU/Swish are smooth ReLU-like curves that slightly dip below zero — GELU is what GPT-2/BERT use inside their FFN layers; SiLU (as part of SwiGLU) powers Llama-family models.

See them in action bending decision boundaries on the MLP page, or continue to attention — where a different non-linearity, softmax, takes center stage.