Lesson 3.3
Activations
The squashing function inside each neuron. It looks like a small detail, but without it a deep network is no better than a single line. This is the ingredient that makes depth worth anything.
Why a squasher at all.
Here's the thing that surprises people: if you remove the activation and just stack weighted sums, the whole deep network collapses into a single weighted sum. A line fed into a line is still a line. No matter how many layers you pile up, without a nonlinearity in between you can only ever draw a straight boundary.
The activation is the kink that breaks that collapse. By bending each neuron's output, it lets stacked layers combine into curves and complex shapes. It is the reason depth buys you anything.
The usual three.
Drag x and watch how each one maps the same input differently:
Drag a dot, one input x, three squashers
drag far out and the dot dims, that's saturation, where the slope is ~0
grab any dot and slide it left or right
Watch the flat tails.
Drag x far out on tanh or sigmoid and the curve goes flat. Flat means slope near zero, which means the local gradient there is tiny, so almost no gradient flows back through a saturated neuron, and learning stalls. This is the vanishing gradient problem. ReLU dodges it on the positive side (constant slope of 1), though a neuron stuck at negative inputs can "die" at zero gradient. You don't need to solve this for a small net; just know the flat regions are where learning goes quiet.
For monktensor's make_moons task, tanh or relu both work fine in the hidden layers. The local gradients you'll need are in the lesson 1.4 table: tanh gives 1 − out², relu gives 1 if the input was positive else 0.
Key takeaways
Without a nonlinearity, stacked layers collapse into a single line; the activation is what makes depth useful.
tanh and sigmoid are S-curves; relu is a simple ramp and the common default.
Saturated (flat) regions have near-zero gradient, so learning stalls there, the vanishing gradient problem.