Lost in the Middle

The first and last items on a long list.

Read someone a grocery list of fifteen items, then ask them to recall it. They will nail the first couple and the last couple, and the ones in the middle turn to mush. People do this. It turns out models do something similar when they read a long prompt.

You have just learned to stuff passages into the prompt and to respect the token budget. The quiet surprise is that where a passage sits in that prompt changes how well the model uses it. The same passage, same words, lands differently depending on its position.

A U-shaped attention.

Plot how reliably a model uses a passage against where that passage sits, and the line is shaped like a U. It is high at the start, dips through the middle, and climbs back up at the end. The model attends strongly to the top and the bottom of what it reads and weakly to whatever is sandwiched in between. A correct, relevant passage parked in the middle can simply get overlooked.

Drag the highlighted key passage to different slots in the prompt below and watch the readout. It scores high near the top and the bottom, and slumps in the middle. The exact shape is illustrative, but the direction is the lesson.

The practical move.

Once you know the shape, the fix is almost free. Put the passage most likely to hold the answer near the top or the bottom of the context, not buried in the stack. If your reranking step gives you a best guess at the most relevant passage (Unit 5 builds that), this is exactly where you spend it.

A common pattern is to order passages by relevance and place the strongest at the two ends, letting the weaker ones fill the middle where attention is thin anyway. You are not adding any tokens or changing the model. You are just being deliberate about seating, putting your best guest where the host will actually notice them.

A worked example.

Retrieval returns six passages and you fit all six in the budget. The one that actually answers the question came back ranked fourth, so by default it lands fourth in the prompt, smack in the middle. The model reads everything, leans on the ends, glides past slot four, and gives a vague or wrong answer even though the right text was right there.

Now reorder: move that fourth passage to the top. Nothing else changes, same six passages, same token cost. The model reads it where attention is high, finds the answer, and cites it. The win came from seating, not from retrieving anything new.

Key takeaways