Lesson 0.5
The Chain Rule
The one rule backpropagation is built on. It tells you the slope of a function-inside-a-function, and it has a second half, what happens when one input feeds two places, that the whole engine hinges on.
Gears.
Gear A drives gear B; gear B drives gear C. If A turns and B spins three times as fast, and B turns and C spins twice as fast, then turning A spins C six times as fast. The rates multiply along the chain. That is the chain rule, and that one sentence is the engine of all of deep learning.
Grab gear A and spin it
B turns 3× as much, C turns 6×, the rates multiply down the chain
C turns 6× as fast as A, because 3 × 2 = 6
Real models are functions stacked on functions stacked on functions, dozens deep. The chain rule is how a nudge at the very end (the loss) gets traced back, layer by layer, to every parameter at the start. Multiply the local rates along the way.
The rule, stated plainly.
Suppose y depends on u, and u depends on x. To find how y moves when x wiggles, multiply the two local rates:
The slope through a chain is the product of the slopes of each link.
"How fast y moves with u" times "how fast u moves with x." The middle quantity cancels the way fractions do, which is a good way to remember it.
A worked chain.
Let u = 2x and y = u². The local rates: du/dx = 2 (from lesson 0.3) and dy/du = 2u. Multiply:
Check it with a nudge at x = 1: there y = (2·1)² = 4. Bump x to 1.001 → u = 2.002 → y = 2.002² ≈ 4.008. The output moved about 0.008 for a 0.001 input nudge: slope ≈ 8. Matches 8x = 8. The gears were right.
The second half: when one input feeds two places.
Here's the part most explanations skip, and it's the one your engine will live or die on. What if x doesn't flow down a single chain, but splits and influences the output through two different routes? Then its total effect is the sum of the effects along each route. Influence through route 1 plus influence through route 2.
When x reaches y through two paths, add the chain-rule product of each path.
The cleanest example: y = x × x. The same x goes into both slots of the multiply. Treat each slot as its own path. Wiggling the first x changes y at rate "the other one," which is x; same for the second slot. Add them:
This is the rule behind gradient accumulation. In the computation graph, whenever a value is used in more than one place, the gradients flowing back from each use must be added together, not overwritten. Forgetting to add is the single most common bug when you write an autograd engine. We give it a full lesson in Unit 2, but the reason is right here: multiple paths sum.
Why this is the whole of backprop.
Put the two halves together and you have backpropagation in one sentence: to get the gradient of the loss for any value, multiply the local rates along each path from that value to the loss, and add across paths. Unit 1 builds the graph that records those paths; Unit 2 walks it backward doing exactly this multiply-and-add. Everything else is plumbing.
Key takeaways
Chain rule: the slope through a chain of functions is the product of the local slopes. Rates multiply.
When one input reaches the output through multiple paths, its gradient is the sum over those paths.
Multiply along paths, add across paths. That sentence is the entire backpropagation algorithm.