Lesson 4.3
Gradient Descent & SGD
The act of learning itself: take the gradient, step the parameters downhill, repeat. Simple to state, but the step size makes or breaks it, and one easy-to-forget reset causes the most common training bug.
Step downhill, repeat.
You have a loss and, thanks to backprop, its gradient for every parameter. The gradient points uphill (lesson 0.4), so to lower the loss you step each parameter a little in the opposite direction. Do it again. And again. That's gradient descent, the entire learning algorithm in one move:
Drag the learning rate and step the ball down the bowl:
Roll downhill on L(w) = w², one step at a time
each step: w ← w − lr × 2w. Small lr crawls; near 1 it overshoots; above 1 it flies apart.
The learning rate is the knob you'll fight most.
The learning rate is the step size, and the demo shows its whole personality. Too small and it crawls, correct direction, glacial progress. Just right and it slides smoothly to the bottom. Too big and it overshoots, bouncing across the valley; past a point it diverges entirely, flying uphill. Most of the practical art of training is finding a learning rate that's brisk but stable.
Zero the gradients first (or else).
Remember from Unit 2 that gradients accumulate (each backward pass adds with +=). So before every backward pass you must reset every gradient to 0. Skip it and this step's gradients pile on top of last step's leftovers, and the model lurches around or stalls. The training loop is: zero the grads, backward, step. Forgetting the zero is one of the most common training bugs there is, and nothing crashes to warn you.
Why "stochastic" (SGD).
Computing the loss over the entire dataset for every step is exact but slow. Stochastic gradient descent uses a small random batch of examples each step instead. The gradient is noisier, but you take far more steps per second, and the noise can even help escape bad spots. For a dataset as small as make_moons you can use all of it each step; the principle scales to the batches big models depend on.
Key takeaways
Gradient descent steps each parameter opposite its gradient: w ← w − lr × grad, repeated.
The learning rate sets the step size: too small crawls, too big overshoots or diverges.
Zero the gradients before each backward pass (they accumulate); SGD uses small batches to step faster.