Lesson 2.2
Codebooks: a vocabulary of vectors
Once you have a set of representative centres, you can store any vector as just the index of its nearest one.
A dictionary of shapes
The centres k-means produces form a codebook: a short, numbered list of representative vectors. Think of it as a dictionary. Instead of writing out a whole vector, you find the closest entry in the dictionary and write down its number. To get the vector back, you look the number up. The vector is never stored, only its index.
Drag the coral point below. It always reports the nearest codebook entry, and that single index, here a number from 0 to 5, is all that gets stored. Reconstruction is just the lookup.
Drag the point; it is stored as the nearest codebook entry
stored as #2 · 1 byte instead of the full vector
reconstruct = look up entry #2
A codebook is a numbered list of representative vectors (the k-means centres). A vector is stored as the index of its nearest entry, and reconstructed by looking that index up.
Why 256 entries is the magic number
How big should the codebook be? The favourite size is 256, and the reason is the byte again. An index into a 256-entry codebook fits in exactly one byte. More entries mean each vector lands closer to its representative, so less error, but the index needs more bits and the codebook itself costs more to store and search. 256 is the sweet spot the classic methods settled on.
There is still a problem hiding here. A real embedding (the vector an AI model produces to stand for something like a word or an image) has hundreds of dimensions, meaning hundreds of separate numbers in the list. Every extra dimension multiplies the number of distinct shapes the data can take, so to cover that space well with a single codebook you would need a dictionary far too large to ever build or store. The next unit is the elegant trick that dodges it.
Key takeaways
A codebook is a numbered list of representative vectors; a vector is stored as its nearest entry's index.
256 entries is the favourite size because the index fits in one byte.
One codebook cannot cover a high-dimensional vector well, which is the problem product quantization solves.