Lesson 0.2
Learning Is Just Fitting a Function
The word 'learning' sounds mysterious. Underneath, it's an ordinary idea: turn some knobs until a machine's answers stop being wrong. Here's that idea, made concrete.
A machine with knobs on the side.
Imagine a machine with a few knobs on it. You feed something in, it spits a number out. Right now the knobs are set randomly, so the output is nonsense. But you have a stack of examples where you know the right answer. So you play a game: feed in an example, look at how far the output is from the right answer, then turn the knobs a little to make it closer. Do that thousands of times and the machine starts producing good answers on its own.
That's the whole of "learning" in this course. The machine is a function. The knobs are its parameters (also called weights). "Learning" is just searching for knob settings that make the function fit your examples. Nothing is being understood or memorized; numbers are being tuned.
Function: a rule that takes an input and produces an output. Parameters: the adjustable numbers inside that rule. A child learning to throw a ball is adjusting parameters too, every miss tells the arm which way to correct next time.
Measuring "wrong" with a single number.
To turn the knobs sensibly, you need to know how wrong you currently are, as one number. That number is the loss. High loss means bad predictions; low loss means good ones. Learning is the hunt for knob settings that make the loss small.
A simple loss: take each example, find the gap between your prediction and the true answer, square it so misses in either direction count as bad, and average over all examples. One example off by 2 contributes 4; off by 0 contributes nothing. Average them, that's your score for the current knob settings.
Fitting a line, by feel.
Say your machine has two knobs, w and b, and predicts with the rule below. The data is a cloud of points you want the line to pass through.
Drag the two ends of the line to fit the dots
the red sticks are the errors; the loss adds up their squares
drag the line down to the dots and watch the loss fall, that downhill is what training does automatically
Dragging the line's two ends is the same as choosing the two knobs: the slope w and the height b. Pick w = 0, b = 0 and the line is flat on the floor, terrible fit, high loss. Nudge w up and the line tilts to follow the points; the loss drops. Overshoot and it climbs again. If you plotted the loss against w, you'd see a valley: learning is walking downhill to the bottom of that valley.
This reframes the scary phrase "the AI learns" into something plain: learning is minimizing a function (the loss) by adjusting its inputs (the parameters). The entire rest of this course is about doing that walk-downhill efficiently, which is exactly what gradients are for.
What learning is NOT.
Not memorizing the examples.
The goal is to do well on inputs it has never seen. A machine that only parrots the training data has failed. We'll return to this as "generalization" in Unit 4.
Not magic, and not search-by-luck.
We don't try random knob settings and hope. We compute exactly which way each knob should move. That's the next three lessons.
Key takeaways
A model is a function with adjustable numbers (parameters). Learning means choosing parameters that fit your examples.
The loss is a single number measuring how wrong the predictions are. Learning is making the loss small.
Training is minimizing a function by adjusting its inputs. Gradients are how we know which way to adjust.