Lesson 1.4
Local Gradients
Each operation knows one small thing: how its own output reacts to each of its own inputs. That little fact, attached to every node, is all backprop needs. The chain rule does the rest.
What "local" means.
A local gradient is an operation's answer to the nudge question about only itself: "if I wiggle one of my inputs a hair, how does my output move?" It ignores the rest of the graph entirely. A multiply node doesn't know or care what happens ten steps downstream, it only knows its own rule.
This is the per-operation derivative from lesson 0.3, now thought of as a property attached to each node. Every operation carries its own little rule.
The rules you'll need.
A scalar neural net needs only a handful. Each row says: given the output's gradient, what's the local factor for each input?
| Operation | Local gradient | In words |
|---|---|---|
| c = a + b | ∂c/∂a = 1, ∂c/∂b = 1 | a sum passes gradient straight through |
| c = a × b | ∂c/∂a = b, ∂c/∂b = a | each input's factor is the other input |
| c = aⁿ | ∂c/∂a = n·aⁿ⁻¹ | the power rule |
| c = tanh(a) | ∂c/∂a = 1 − c² | flat when saturated, steep near 0 |
| c = relu(a) | ∂c/∂a = 1 if a>0 else 0 | on or off, like a gate |
Notice the multiply and add rules are exactly what we found by hand in lessons 0.1 and 0.5. Nothing new, just collected in one place and labeled "local."
A worked node.
Take the multiply d = a × b with a = 2, b = -3. Its local gradients:
One operation, up close, drag the inputs
the local gradient is just "if I nudge this input, how fast does the output move?", ignoring the rest of any graph
The node can produce these from the values it already remembered during the forward pass. It needs nothing else.
The design this points to.
Every node stores two things from the forward pass: its output value and enough to compute its local gradients. Backprop then chains these together, multiply local gradients along each edge, add where paths merge (lesson 0.5). That single idea is the entire engine you'll build. The next unit turns "chain the locals together" into a concrete walk over the graph.
Key takeaways
A local gradient is one operation's "if my input wiggles, how does my output move?", ignoring the rest of the graph.
You only need a handful of rules: add, multiply, power, and a couple of activations.
Each node stores its value and local gradients; backprop chains them with multiply-along, add-across.