Lesson 2.1

The Backward Pass

Now the other direction. Start at the output, seed its gradient at 1, and walk the graph in reverse, letting each node hand gradient to its inputs. By the end, every value knows how it affects the output.

math 11 min read

Tracing the influence home.

The forward pass asked "what's the output?" The backward pass asks the reverse, for every value at once: "how much does the output change if this value wiggles?" That answer, stored on each node, is its gradient.

It starts with a trivial fact: the output's effect on itself is 1 (wiggle the output by a sliver and it changes by exactly that sliver). So we seed the output's gradient at 1, then walk backward. At each node we take the gradient sitting on its output and use the node's local gradient rule to push the right amount onto each input. That "push" is one multiply, the chain rule, one edge at a time.

A full backward pass, by hand.

Reuse e = (a × b) + c with a = 2, b = -3, c = 10, so d = -6 and e = 4. Walk it backward:

e.grad = 1  seed: the output's effect on itself
add e = d + c → pass straight through:  d.grad = 1,  c.grad = 1
mul d = a × b → each gets the other × incoming:  a.grad = b × 1 = -3,  b.grad = a × 1 = 2

Final gradients:

a.grad = -3  ·  b.grad = 2  ·  c.grad = 1

The backward pass of e = (a × b) + c

seed the output at 1, then let the gradient flow backward

a grad 0 b grad 0 c grad 0 × d grad 0 + e grad 0

each node hands its inputs (their local gradient) × (the gradient that arrived), multiply along, add across

Sanity check c: raising it by 1 raises e = d + c by exactly 1, so c.grad = 1. Correct. And a.grad = -3 says nudging a up a hair pulls e down at rate 3.

Each node is selfish, and that's enough.

No node needs the big picture. Each one only needs the gradient arriving at its output and its own local rule (lesson 1.4). It multiplies, pushes the result to its inputs, and is done. Do that for every node in the right order and the chain rule assembles itself across the whole graph. That "right order" and the "what if a node feeds two places" wrinkle are the next two lessons.

Key takeaways

1

The backward pass seeds the output gradient at 1, then walks the graph in reverse computing every node's gradient.

2

Each node uses only the gradient at its output times its local rule to push gradient to its inputs.

3

Do that locally for every node and the chain rule assembles itself across the whole graph.