Backpropagation, by assigning blame
A click model predicted 0.90 for an impression that wasn't clicked. The loss is high, and the network has 9 parameters. How much is each one to blame, and can you find out without trying each one?
This lesson follows real numbers forward into the loss, then traces the blame back through every weight and every ReLU, one step at a time. Three kinds of quantity stay visually distinct: activations (values computed going forward), parameters (the weights being learned), and gradients (blame flowing backward).
Why not just nudge each weight?
A gradient answers a concrete question: if I move this one parameter a tiny bit, how much does the loss move?
Change in loss over change in weight: \(0.003 / 0.01 = 0.3\). It's positive, so gradient descent lowers this weight. That's a finite-difference estimate, and it's how we'll check backprop later.
One nudge per parameter, so a billion passes per update. Backpropagation gets all of them from one forward pass plus one backward pass, costing roughly 2 to 3 forward passes in total. It does this by reusing work: the blame arriving at a node is computed once and shared by every weight behind it. The rest of this lesson is that reuse, made visible.
Follow the numbers into the loss
Two features describe the impression: \(x_1 = 1.0\), the user has clicked this creator before, and \(x_2 = 0.6\), the item's normalized popularity. Two ReLU hidden units feed a sigmoid output. The impression was not clicked, \(y = 0\).
Click any node to see its computation. Click any weight to see what it is.
ReLU outputs \(\max(0, -0.9) = 0\). For this example, unit 2 is off. Keep that in mind: it decides what happens to its blame in the backward pass.
The output is \(p = \sigma(1.5 \cdot 1.40 + 0.8 \cdot 0 + 0.1) = \sigma(2.2) = 0.90\), and the loss is \(-\ln(1 - 0.90) = 2.305\). The model was confident and wrong.
The blame game
Work backward from the loss. At each step, predict how much blame arrives, then watch it appear on the diagram. One rule does all the work, the chain rule: blame arriving at a quantity equals the blame at whatever it feeds, times how strongly it feeds it.
Check backprop against nudging
Backprop and nudging must agree. Pick any parameter: the page nudges it by ±0.0001, reruns the forward pass, and compares the finite-difference slope with the backprop gradient.
This is how you debug a hand-written backward pass. A mismatch in one parameter points to the exact line that's wrong. In practice you compare relative error, \(|g_{\text{bp}} - g_{\text{fd}}| / \max(|g_{\text{bp}}|, |g_{\text{fd}}|)\), in float64. Around \(10^{-7}\) is fine, and \(10^{-2}\) is a bug. Avoid the kink of ReLU at exactly 0, where the slope isn't defined.
Why a sigmoid output pairs with cross-entropy
In section 3 the output blame was exactly \(p - y\). The sigmoid's own slope \(p(1-p)\) cancelled against the derivative of the log. That cancellation depends on the loss.
\(\frac{\partial L}{\partial z} = 2(p-y)\cdot p(1-p) = 2(0.999)(0.001) \approx 0.002\). The sigmoid is saturated, and its tiny slope multiplies the blame away. The model is as wrong as it can be, and it barely learns. With cross-entropy the blame is \(p - y = 0.999\).
The same thing happens inside a network. Every saturated sigmoid or tanh on the path multiplies the blame by a slope near 0. Stack enough of them and the early layers stop learning: that's the vanishing gradient. ReLU's slope is exactly 1 when it's on, which is a big part of why it replaced sigmoid in hidden layers. Its slope is exactly 0 when it's off, which is the "dead ReLU" risk you saw with unit 2.
Apply it somewhere else
When a quantity feeds the loss along several paths, the total derivative is the sum over paths (multivariable chain rule). Frameworks do this automatically by accumulating into .grad. That's also why forgetting zero_grad() silently sums gradients across steps.
Identical units compute identical activations, so they receive identical blame and identical updates, forever. The layer behaves like a single unit. Random initialization breaks the symmetry. Its scale (Xavier or He) keeps activations and gradients from shrinking or blowing up with depth.
A weight's gradient is (blame at its output) × (its input), and the input is 0. Encoding "missing" as 0 makes it indistinguishable from a real zero, and the weights never learn anything about the missing case. Add an explicit missing indicator, or use a learned default embedding.
The gradient at layer 1 is a product of about 50 Jacobians. If their typical scale is below 1, it vanishes geometrically, and above 1 it explodes. Residual connections add an identity path: \(\partial(x + f(x))/\partial x = I + f'(x)\), so blame can flow back unattenuated. Normalization and careful initialization help too.
Write the backward pass
Plain editor, no completion, hidden tests. Batch version of this lesson's network. State the shapes out loud first. ⌘/Ctrl+Enter runs.
Exercise 1: forward and backward for a 2-layer MLP
Exercise 2: a numerical gradient checker
Answer out loud, then check
- Reverse mode computes the gradient of one scalar output with respect to all inputs in a single backward pass. Forward mode needs one pass per input direction.
- Training has one scalar loss and billions of parameters, so reverse mode is the right trade: cost is about 2 to 3× a forward pass.
- Forward mode wins when outputs outnumber inputs, for example Jacobian-vector products with few inputs.
- The activations each layer's weight gradient needs (its inputs), plus masks such as ReLU gates. This scales with batch size × sequence length × depth × width, and often dominates parameter memory.
- Activation checkpointing stores only some layers' activations and recomputes the rest during the backward pass (about 30% more compute for large memory savings). Other levers: mixed precision, smaller micro-batches with gradient accumulation, and fused kernels.
- Detect: for each unit, the fraction of training examples where it's active. A unit active on 0% gets zero gradient forever.
- Causes: a large update pushes the bias or weights so that the pre-activation is negative for all inputs. High learning rates are the usual trigger.
- Fixes: lower learning rate or warmup, better initialization, LeakyReLU or GELU (nonzero slope for negatives), normalization before the activation.
BCEWithLogits instead of sigmoid followed by BCE?- Numerical stability: \(\log(1 - \sigma(z))\) underflows to \(\log 0\) for large \(z\). The fused form computes \(\operatorname{softplus}(z) - yz\) stably.
- The fused gradient is exactly \(\sigma(z) - y\), with no saturated slope to multiply through.
- Same reason cross-entropy takes logits and uses log-softmax internally (lesson 1).
- Backprop multiplies local Jacobians layer by layer (or step by step in an RNN). A product of many factors with norms below or above 1 shrinks or grows geometrically.
- Fixes: residual connections (identity path), normalization layers (LayerNorm or BatchNorm), variance-preserving initialization, gated recurrences (LSTM/GRU), and gradient clipping for explosions.
- Diagnose by logging per-layer gradient norms, not just the loss.
Sources: Rumelhart, Hinton & Williams, Learning representations by back-propagating errors (1986); Glorot & Bengio (2010) and He et al. (2015) on initialization; He et al., Deep Residual Learning (2015); Chen et al., Training Deep Nets with Sublinear Memory Cost (2016).