Lesson 1.3
Parameters vs Activations
Not every number in the graph is the same kind of thing. Some you adjust to learn; some are just computed and thrown away. Mixing them up is a top source of bugs, so let's pin it down now.
Knobs and readouts.
On a machine, there are knobs you turn and readouts that just display whatever the knobs produce. You don't "set" a readout, it shows you a consequence. The graph has the same split:
Parameters (the knobs)
Weights and biases. The numbers you adjust to make the model better. These are leaves of the graph, and they're what learning actually changes.
Activations (the readouts)
The intermediate values computed during the forward pass. You don't set them; they fall out of the parameters and the data. They're recomputed from scratch every forward pass.
There's a third kind of leaf: the data (your inputs). Like parameters, data sits at the leaves, but unlike parameters, you never change it. It's given.
Turn the knobs, the gauges follow
knobs you turn
parameters (weights)
drag a knob up/down
gauges that follow
activations (computed)
data: x1=1.5, x2=-1 (fixed)
backprop gives every value a gradient, but a training step only turns the knobs
Everyone gets a gradient; only knobs get turned.
Here's the subtlety that bites people. Backprop computes a gradient for every node in the graph, parameters, activations, even the data. That's just how the chain rule flows. But the training step only updates the parameters. The activations' gradients are used in passing (to push gradient further back) and then discarded; the data's gradient is usually ignored entirely.
Say d = a × b, where a is a parameter (a knob) and b is data. Backprop will compute a gradient for both. But your optimizer nudges only a; b stays exactly as the dataset gave it. When you build the engine, this is the line between "things in the parameter list" and everything else.
Why frameworks "mark" the leaves.
Because of this split, real frameworks let you flag which leaves are learnable (PyTorch calls it requires_grad, and gathers them via parameters()). The flag answers one question: "when I take a training step, which numbers am I allowed to change?" In your engine, the network's parameters() list will be exactly that set, the knobs, and nothing else.
Key takeaways
Parameters are the knobs you adjust (weights, biases); activations are computed readouts; data is a fixed leaf.
Backprop computes a gradient for every node, but the optimizer updates only the parameters.
Your model's parameter list is precisely the set of learnable leaves, keep it distinct from everything else.