Lesson 4.4
Initialization & Regularization
Two short but decisive choices: where the knobs start, and how to stop the network from just memorizing. Get these wrong and a perfectly correct engine still won't learn well.
Where the knobs start matters.
Before the first step, every weight needs a starting value. The tempting choice, set them all to zero (or all the same), quietly breaks the network. If every neuron in a layer starts identical, they all compute the same thing, all receive the same gradient, and so update identically forever. They stay clones; the layer never spreads out to learn different features. This is the symmetry problem.
Four neurons, do they ever become different?
all equal: identical gradients, so they move together and stay clones, the layer never spreads out
The fix is to start weights small and random. Random breaks the symmetry so neurons can specialize; small keeps the early activations away from the flat, saturated tails (lesson 3.3) where gradients vanish. Too large and neurons saturate or the loss explodes from step one.
Stop it from memorizing.
A network with enough knobs can memorize the training points exactly, drawing a wildly wiggly boundary that threads every dot, and then fail on new points. That's overfitting: great on training data, poor on the held-out set. The tell is the validation loss creeping up while the training loss keeps falling.
Regularization pushes back. The common form, L2 (weight decay), adds a small penalty for large weights to the loss. The network now has to justify big weights with real reduction in error, so it prefers smaller weights and smoother, simpler boundaries, the kind that generalize.
For make_moons: start weights small and random, and add a little L2. That combination gives a network that actually learns (symmetry broken, gradients flowing) and draws a clean curve between the moons instead of a jagged line that memorizes the dots.
Key takeaways
Don't start all weights equal, neurons would stay clones (the symmetry problem). Start small and random.
Overfitting is memorizing the training data; the sign is validation loss rising while training loss falls.
L2 regularization penalizes large weights, favoring smoother boundaries that generalize.