The Context Budget

A suitcase, not a moving truck.

You cannot pour every passage you retrieved into the prompt. The prompt has a fixed size, like a suitcase. Pack carefully and the essentials fit. Try to cram in everything and the zipper will not close. In the last lesson you learned how to build the prompt. This one is about the wall you hit: it can only hold so much.

That wall is the context window, the maximum amount of text a model can read at once. Everything you send has to fit inside it: the instruction, the passages, the question, and the answer the model writes back. Go over and the request simply will not run.

What a token is.

The window is not measured in words or characters. It is measured in tokens. A token is a small chunk of text, roughly a word or a piece of a word. Common short words are usually one token; longer or rarer words get split into two or three. A loose rule of thumb in English is that a token is about three-quarters of a word, so 100 tokens land near 75 words.

The number is approximate and depends on the model's tokenizer, the rulebook that does the splitting. For budgeting you do not need the exact count. You need a feel for the trade: more text in means more tokens spent, and the window holds only so many.

The usable budget is smaller than the window.

Here is the catch people miss. The window is not all yours for passages. The question takes some. The instruction takes some. And crucially, the model needs room to write the answer, since the reply lives in the same window as the input. So the budget for passages is the window minus the question minus the instruction minus the space you reserve for the answer.

Reserve too little for the answer and the model gets cut off mid-sentence. Reserve too much and you waste room you could have spent on a relevant passage. The usable passage budget is what is left after everyone else takes their slice.

A worked example (numbers illustrative).

Say your model has a window of 8,000 tokens. The question and instruction together cost about 200 tokens. You want room for a thorough answer, so you reserve 800 tokens for the reply. That leaves 8,000 minus 200 minus 800, which is 7,000 tokens for passages.

Now suppose each retrieved passage runs about 350 tokens. How many fit? 7,000 divided by 350 is 20 passages. That is your ceiling for this question. Retrieval might have returned 50 candidates, but only the top 20 will fit, so order and relevance suddenly matter a lot.

The chunk-size tension, back again.

This is where chunking comes home to roost (lesson 1.3, where you split a document into cards). Big chunks carry more context per passage but cost more tokens each, so fewer fit in the budget. Small chunks are cheap, so many fit, but each one carries less surrounding context and might lose the thread.

There is no free lunch. A bigger chunk and fitting more chunks are pulling on the same rope, the token budget. The right chunk size is the one that puts the most relevant material in the window, not the most material. The budget is the constraint that forces you to choose.

Key takeaways