Lesson 1.2
Calibrating the range
The 256 codes are only as good as the min and max you give them. Too wide wastes them; too narrow throws data away.
Where to put the fences
The 256 codes are spread evenly between your chosen min and max. Choosing those two fences well (the step called calibration) is the only real decision in int8. Set them too far apart and most codes land in empty space the data never visits, so the steps near your actual values are needlessly coarse. Set them too close and the outliers (the few values sitting far from the rest) fall outside the fence and get clipped, flattened onto the edge and losing their real position.
Drag the two window edges below. Green points are quantized cleanly; a point turns red the moment it falls outside and clips. Watch the average error as you go too wide, then too narrow.
Drag the window edges to calibrate the range
window [-2.0, 6.0] · clipped 0 · avg error 0.004
Calibration is choosing min and max. Too wide spends codes on empty range; too narrow clips outliers. The best window just covers the real spread of the data.
One window, or many
A single min and max for a whole vector is the simplest choice, but you can do better by calibrating in smaller groups. Give each dimension its own min and max, or each vector its own, and the fences hug the data more tightly, shrinking the error. The cost is bookkeeping: every extra min and max is itself a number you have to store alongside the codes (this housekeeping data is called metadata), and it eats into the compression. Tighter ranges, more metadata, the same trade-off in miniature.
Work it out
A vector's values run from -0.8 to 0.9, but you calibrate with a min of -5 and max of 5 "to be safe." Roughly what fraction of the 256 codes ever get used, and what does that do to the error compared with a tight range?
Key takeaways
int8's only real decision is calibration: where to place the min and max fences.
Too wide wastes codes on empty range; too narrow clips outliers onto the edge.
Per-dimension or per-vector ranges cut error but add metadata, the same trade in miniature.