Lesson 0.4
Gradients: Derivatives With Many Knobs
A derivative handles one knob. A real model has thousands. The gradient is just the whole list of derivatives at once, and it points the way uphill, so learning walks the other way.
The hill, now in many directions.
Back on the foggy hill, but now you can step in any direction, not just east-west. The steepness depends on which way you face. So a single slope isn't enough, you need a slope per direction. Collect the slope along each axis and you get an arrow that points straight uphill, as steep as the hill gets. That arrow is the gradient.
A model has many parameters, so its loss is a landscape in many dimensions, not a 2D curve. You can't picture it, but the math is the same: the gradient is the list of "which way and how hard" for every knob, bundled into one vector.
Partial derivatives: one knob at a time.
To find the gradient, you ask the nudge question once per knob, holding the others still. "If I wiggle only w₁ and freeze everything else, how does the loss move?" That's a partial derivative, written ∂L/∂w₁. Do it for every knob and stack the answers:
The gradient of the loss: one partial derivative per parameter.
Each entry is just an ordinary derivative, the thing from lesson 0.3, taken with respect to one parameter. Nothing new, only bookkeeping: many derivatives kept in a list.
Uphill, so walk downhill.
The gradient points toward increasing loss, the steepest way up. We want loss to go down. So learning takes a small step in the opposite direction of the gradient. That single rule is gradient descent, which gets its own lesson in Unit 4:
Each parameter steps a little, against its own gradient. The step size is the learning rate.
A worked example with two knobs.
Let the loss be L(a, b) = a² + b², a bowl with its lowest point at the origin. The partials: wiggling a changes a² at rate 2a (the b² term is frozen, contributes 0), and likewise for b:
Drag the ball around the hill
the red arrow points straight uphill, learning steps the opposite way
uphill = the gradient; downhill = where a training step moves you
That arrow (6, 8) points away from the origin, uphill, out of the bowl. Step the other way and both a and b shrink toward 0, the bottom of the bowl, the smallest loss. Repeat the step and you slide to the minimum. That slide is training.
A network with a million parameters has a million-entry gradient. Computing it one knob at a time with finite differences would be hopeless. The whole reason backpropagation exists is to compute this entire vector in one efficient sweep, the punchline of Unit 2.
Key takeaways
A gradient is just a list of derivatives, one per parameter (each a partial derivative: nudge one knob, freeze the rest).
It points uphill (toward higher loss), so learning steps in the opposite direction, downhill.
Backprop's job is to produce this whole vector cheaply, no matter how many parameters there are.