Lesson 1.3
Chunking a Document
A document is too big to embed as one blob, so you split it into passages. How big each piece is changes what you can find.
Tearing a book into index cards.
Imagine answering questions about a whole book by writing one summary card for the entire thing. Ask "what did the author say about the harbor on page 200?" and your card, stretched to cover all four hundred pages, gives you a blur. The fix is obvious: tear the book into index cards, each holding one small idea, so a question can land on the exact card it needs.
Embeddings have the same problem. One embedding is a single point of meaning, so embedding an entire document smears every topic in it into one averaged point that is precise about nothing. The answer is to chunk the document: split it into passages, embed each one separately, and let a question match the passage it actually wants.
Drag the chunk size and watch the split change.
The one knob that matters most is how big each piece is, the chunk size (measured in words or tokens). Grab the handle below and slide it. The document re-splits live, the chunk count updates, and the note tells you what you are trading away at each end.
Drag the handle to set the chunk size
Chunks: 3
Slide it small and you get many narrow cards: a question lands on a precise one, but each card is so short it can lose the thread of what came before it. Slide it large and you get a few fat cards: each keeps plenty of context, but the matching passage drags in a lot of unrelated text, and that text costs you later when it has to fit in the prompt.
The tradeoff, and the overlap fix.
There is no universally right size; it is a balance. Too big means imprecise retrieval (the right answer is buried in a wall of off-topic text) and wasted room in the prompt later. Too small means a passage can lose the surrounding meaning: split mid-thought and "it doubled the next year" lands on a card that never says what "it" was.
A common cushion is overlap: let each chunk repeat the last few words of the one before it. A sentence cut by a boundary then survives on at least one card whole, so an idea is not lost at the seam. The cost is a little duplication, paid to avoid losing meaning at the edges.
small chunks → precise match, but a passage can lose its context.
large chunks → rich context, but imprecise retrieval and wasted prompt room.
overlap → repeat a few words across the seam so ideas survive the cut.
Chunk size is a dial between precision and context. Small cards are precise but forgetful; large cards remember but blur. You tune it for your documents, and a little overlap saves the ideas that fall on a boundary.
What chunking is not
A summary step
Chunking does not rewrite or shorten the text. It cuts the original into pieces and keeps every word. A question still reads the real passage, not a paraphrase of it.
A free choice
The size is not cosmetic. It sets what retrieval can find and how much you pay in the prompt later. Picking it badly quietly caps how good the whole system can get.
Key takeaways
A document is split into passages (chunks) and each chunk is embedded on its own, so a question can match the exact piece.
Chunk size trades precision against context: too big is imprecise and wasteful, too small loses surrounding meaning.
A little overlap between chunks keeps an idea from being lost at a boundary, at the cost of small duplication.