Lesson 0.3
Derivatives, Intuitively
A derivative is one number that answers the nudge question from lesson 0.1: wiggle the input a hair, how much does the output move? No heavy calculus, just the few ideas the framework needs.
Standing on a hill in the fog.
You're on a hillside, fog everywhere, can't see a thing. But you can feel the ground under your feet tilt. That tilt tells you two things: which way is downhill, and how steep it is. That's a derivative, the steepness of a function at the exact spot you're standing.
More precisely: the derivative of a function at a point is how much the output changes per tiny change in the input. If a small step right makes the output rise twice as fast as the step, the derivative is 2. If the output drops, the derivative is negative.
Two pieces of information live in one number. The sign says direction (does the output go up or down when the input goes up?). The size says how fast. Learning uses both: sign tells a knob which way to turn, size tells it how hard.
The nudge definition.
Take a function f. Pick an input x. Add a tiny amount ε (epsilon, think 0.0001). See how much the output moved, and divide by the size of the nudge:
As the nudge ε shrinks toward zero, this ratio settles on the derivative.
That's it. "Output change divided by input change," with the input change made tiny. The exact derivative is what this ratio approaches as ε goes to zero. We write it df/dx or f′(x).
Computing the ratio with a small-but-real ε is called a finite difference. It's approximate, but it's something you can do with arithmetic alone, which makes it the perfect way to check the exact derivatives your framework computes. We'll use that trick to prove the engine correct in Unit 2.
A worked example.
Take f(x) = x² at x = 3. So f(3) = 9. Nudge with ε = 0.001:
slope ≈ (9.006001 − 9) / 0.001 = 6.001 ≈ 6
Drag the dot along the curve x²
the straight line is the slope right where you are
flat at the bottom (slope 0), steeper and steeper as you drag outward, the slope here is 2x
The derivative of x² at 3 is 6. (The exact rule is 2x, and 2×3 = 6, the finite difference landed right next to it, off only by the leftover ε.) That's the nudge question from lesson 0.1, answered with arithmetic.
The handful of rules we'll actually need.
This isn't a calculus course. For a scalar engine you only need a few derivative rules, and each one is the answer to "if I wiggle the input, how does this operation's output move?"
| Operation | Derivative | In words |
|---|---|---|
| c (a constant) | 0 | a constant never moves |
| x | 1 | moves exactly as fast as itself |
| x² | 2x | steeper the further out you are |
| c × x | c | scaling the input scales the slope |
| f + g | f′ + g′ | slopes of a sum just add |
The one rule left, how to handle a function inside a function, is the chain rule, and it's big enough to get its own lesson (0.5). With these plus the chain rule, you can differentiate everything a small neural net needs.
Try it by hand
Estimate the slope of f(x) = x² at x = 2 using a nudge of ε = 0.001. Compute f(2.001), subtract f(2), divide by 0.001. You should land right next to 4 (because the rule 2x gives 2×2 = 4).
Key takeaways
A derivative is output-change per tiny input-change: which way the output moves, and how fast.
You can estimate one with arithmetic (a finite difference), which is how we'll later check the framework's exact answers.
A scalar engine needs only a few derivative rules plus the chain rule. That's the whole math toolkit.