Lesson 2.2
Fan-out & Accumulation
When one value feeds more than one place, gradient comes back from each. The rule is simple but easy to get wrong: add them up. This is the single most common bug when people write an autograd engine.
One value, many uses.
So far each value fed exactly one operation. But values get reused all the time, the same weight feeds many neurons, the same activation feeds several downstream ops. When a value fans out to several places, each place sends a gradient back to it during the backward pass.
What do you do with several incoming gradients? You add them. This is the second half of the chain rule from lesson 0.5: when a value influences the output through multiple paths, its total gradient is the sum over those paths.
The cleanest case: x used twice.
Take y = x × x. The same x goes into both slots of the multiply. During backward, each slot sends a gradient of x back (the multiply rule: each input gets the other). Add them: x.grad = x + x = 2x, exactly the derivative of x². Run it:
Backward pass of y = x × x
x feeds two slots, so two gradients (each = x) come back to it
add both paths → 2x, the right answer. Overwrite and the second path replaces the first, a path is lost.
If you had overwritten instead of added, second gradient replacing the first, you'd get x, which is wrong. The two paths must accumulate.
The bug, and the fix.
When you build the engine: start every node's gradient at 0, and have each consumer add its contribution (+=), never assign (=). Use = and a value used twice will keep only the last gradient, silently halving (or worse) the truth. The symptom is nasty: the network sort-of trains but never quite works, and nothing crashes. Accumulate, always.
Why this is correct, not a hack.
It isn't a fudge to make numbers work, it's the multivariable chain rule. If x reaches the output along two independent routes, nudging x moves the output once through each route, and the total movement is the sum. Adding the gradients is just bookkeeping that total honestly.
Key takeaways
When a value feeds multiple operations, gradient flows back from each, and you add them.
Start gradients at 0 and accumulate with +=, never overwrite. This is the classic autograd bug.
It's the multivariable chain rule: multiple paths to the output mean the gradients sum.