Lesson 4.2
Loss Functions
The loss turns 'how wrong is the model' into one number to minimize. Different tasks call for different losses, but they all do the same job: define what 'good' means so gradient descent has something to chase.
One number that defines "good."
From lesson 0.2: the loss scores how wrong the predictions are, as a single number, so learning has a target to push down. The loss defines what you're optimizing, pick the wrong loss and the model cheerfully gets good at the wrong thing. Crucially, the loss is just one more node at the end of the graph, so backprop flows through it like any other operation.
A few common ones.
Mean squared error (MSE), for predicting numbers
Take the gap between prediction and truth, square it, average over examples. Squaring makes big misses hurt much more than small ones and keeps the loss smooth. Used for regression (predicting a quantity), not our task, but the simplest to picture.
Hinge / max-margin loss, for classification
Wants the correct class to win by a clear margin, not just barely. If the right answer already wins comfortably, no penalty; otherwise, penalty proportional to how short it falls. This is the loss micrograd's make_moons demo uses.
Cross-entropy, for probabilities
Treats the output as a probability and punishes confident wrong answers hardest. The standard loss for modern classifiers. For two classes it's "binary cross-entropy."
Why squaring, why a margin, why probabilities? Each shapes what the model is pushed toward: MSE toward small numeric error, hinge toward confident separation, cross-entropy toward calibrated probabilities. Same machinery, different definition of "good."
Drag the prediction, truth is fixed at +1
each loss says "how wrong" with a different shape; all hit zero at the right answer
the green dashed line is the right answer, drag a dot toward it and every penalty falls to 0
For make_moons.
For monktensor's two-moons task, either the hinge / max-margin loss (like the micrograd demo, usually with a little L2 regularization) or binary cross-entropy works well. Pick one; it becomes the root node of your graph, and backward() from there fills in every parameter's gradient. The choice of loss is a design decision, not a code change to the engine.
Key takeaways
The loss is one number scoring wrongness; minimizing it is the whole goal of training.
MSE fits numbers; hinge and cross-entropy fit classes. The loss defines what "good" means.
The loss is the final node of the graph, so backprop flows through it; for make_moons, hinge or cross-entropy both work.