Lesson 1.1
A range into 256 levels
The simplest quantizer there is: take the range a number lives in, and map it onto 256 evenly spaced integer codes.
One number, one byte
Scalar quantization, also called int8 because each number ends up stored in a single 8-bit byte, is the hello world of compression. You decide the lowest and highest value a number can take, slice that range into 256 evenly spaced steps, and store the number as the index of its nearest step: an integer from 0 to 255. That index fits in a single byte, where the original 32-bit float took four. A clean 4 times smaller.
Drag the value below. The coral dot is the real number; the readout shows the byte code it becomes and the slightly different number you get back when you decode.
Drag the value, read its byte code
value 3.00 → code 159 → decoded 2.99
error 0.012 · step = (6 − (−2)) / 255 = 0.031
int8 quantization stores a float as the nearest of 256 evenly spaced codes (one byte). Decoding turns the code back into a float close to, but rarely exactly, the original.
The two lines of arithmetic
The whole method is two formulas. First, the size of one step (called the scale), which is the range divided by 255. Why 255 and not 256? The codes are numbered 0 to 255, so there are 255 gaps between the first code and the last, and the range has to stretch across those gaps. Then encoding is "how many steps from the bottom," rounded; decoding walks that many steps back up from the bottom.
code = round((x − min) / scale)
decoded = min + code × scale
encode to a byte code, then decode back to a float
The decoded value differs from the original by at most half a step, so the error is bounded by scale / 2. Narrow the range and the step shrinks, and so does the worst-case error. That single observation is the entire next lesson.
What it is not
int8 is not "use 8-bit numbers" in the vague sense. It is a specific, reversible mapping with a chosen min and max. Get the min and max wrong and the codes are wasted or the data is clipped (any value past the chosen edge gets frozen at the boundary, losing its real position), even though every number is still technically one byte. The bits are only as good as the range you picked for them.
Key takeaways
int8 maps a chosen range onto 256 integer codes; each number becomes one byte, a 4x saving over a float.
It is two formulas: a scale of (max - min) / 255, then round to encode and step back to decode.
The worst-case error is half a step, so a tighter range means a smaller error.