Lesson 0.2
Memory vs accuracy
Every bit you add buys accuracy and costs memory. Quantization is the art of choosing where on that curve to sit.
The two dials move together
Picture sketching a smooth hill using only graph-paper steps. With a few coarse steps the sketch is blocky and wrong in places, but it takes almost no ink. Add finer steps and the sketch hugs the hill, at the cost of more ink. The hill is the real data you are trying to capture (its true shape, called the signal), the steps are the levels, and the ink is memory.
Drag the knob below from 1 bit up to 8. Watch the stepped red line close in on the smooth grey one as the error gauge falls, while the memory gauge climbs. The two move in opposite directions, always.
Drag the knob to change how many bits
4 bits = 16 levels · error 0.03
Bits trade directly against error. Spending more bits lowers the average quantization error but raises the memory cost. There is no setting that wins both.
Why the gains taper off
Going from 1 bit to 2 bits is dramatic: the error roughly halves. Going from 7 bits to 8 barely moves the line, because the steps are already finer than the signal's own wobble. This is the shape of almost every quantization curve: big wins early, then diminishing returns. The skill is stopping at the point where one more bit no longer earns its keep.
7 → 8 bits: error barely changes
This is the whole game
Every method in this course, from the simple int8 quantizer to the 2026 research, is a different answer to one question: for a given memory budget, how small can you make the error? A better method is one whose curve sits lower, more accuracy at the same number of bits. To compare them fairly we need a precise way to measure both axes, which is the next lesson.
Key takeaways
Memory and accuracy are a trade: more bits lower the error and raise the cost, never both at once.
The error curve shows big wins for the first few bits, then diminishing returns.
A better quantizer is one whose curve sits lower: less error for the same memory.