Lesson 2.4
Gradient Checking (and why not finite differences)
How do you know your backprop is right? Check it against the slow, simple method from lesson 0.3. And that same comparison answers a deeper question: why we use backprop to train at all.
Two ways to get a gradient.
You now have a fast, exact way to get gradients: backprop. But fast and exact code can still have bugs. So you keep a second, dead-simple way around to check it, the finite difference from lesson 0.3. Nudge one parameter a tiny bit, see how the loss moves, divide:
The two-sided finite difference: a slightly more accurate nudge, used only to check.
Run both. If your backprop says a parameter's gradient is -3 and the nudge says -2.9998, you're correct (the tiny gap is just ε rounding). If they disagree by a lot, you have a bug, almost always a wrong local gradient or a missing accumulation.
This is your most important test.
A gradient check is the one test that proves the heart of the engine works. Build a small expression, compute gradients with backprop, compute them again by nudging, and assert they match within a small tolerance. If that test passes, your autograd is trustworthy. Write it early; lean on it hard.
So why not just train with finite differences?
If nudging gives the gradient and it's so simple, why bother with backprop at all? Cost. To get the gradient for one parameter by nudging, you run a whole forward pass (twice, for the two-sided version). To get gradients for all parameters, you'd repeat that for every single one.
Finite differences
About N forward passes for N parameters. A million parameters means a million forward passes per step. Hopeless.
Backpropagation
One backward pass produces all N gradients, at roughly the cost of one forward pass. Total: about 2× a forward pass, no matter how many parameters.
That gap, N passes versus one, is the entire reason automatic differentiation exists, and why training billion-parameter models is even thinkable. Finite differences are the honest, slow ruler you measure against; backprop is the thing you actually run.
Do they agree? And what do they cost?
the nudge (finite difference) converges on the exact answer, that's how you trust the engine
backprop, 1 pass
nudging, 1 pass per parameter
a million parameters → a million forward passes for nudging, but still one backward pass. That gap is why autograd exists.
Check it by hand
For e = (a × b) + c at a = 2, b = -3, c = 10, backprop gave a.grad = -3. Verify by nudging: compute e with a = 2.001 and again with a = 1.999, then apply the two-sided formula. You should land right next to -3.
Key takeaways
Gradient checking compares backprop against a finite-difference nudge; they must match within a small tolerance.
It's the single most important test of your engine, write it early and trust it.
We don't train with finite differences because they cost ~N forward passes; backprop gets all N gradients in one.