Why Retrieval Isn't Enough

The librarian who only reads the spines.

Picture asking a librarian for the one book that answers your question, and they have a million books. They cannot open and read each one, that would take all day. So they glance at the spines, grab the fifty that look about right, and hand you the stack roughly in the order they noticed them. Useful, fast, and a little sloppy. The book you actually need might be the twentieth one in the pile, not the first.

That is what the first pass of retrieval does. It is built for speed across a huge pile of passages, so it settles for a rough ordering instead of a careful one. The right answer is usually somewhere near the top, but "near the top" is not the same as "at the top."

It scores each passage on its own.

Here is the deeper reason the order is rough. The fast search uses what is called a bi-encoder: it turns the question into a vector (a list of numbers that stands for its meaning) and turns each passage into its own vector, separately. Then it ranks passages by how close their vector sits to the question's. Two encoders, run apart, never looking at each other.

Doing it separately is exactly why it is cheap. The passage vectors are computed once, ahead of time, and stored. When a question comes in you only have to encode the question and measure distances. Nothing about the passages has to run again. That precomputation is the whole trick that makes search fast.

But it comes at a cost: the model never sees the question and a passage at the same moment. It cannot notice that this particular passage answers this particular question, only that their two summaries land near each other in meaning. A passage can summarize close to your question and still not answer it, and a passage can answer it perfectly while its summary lands a bit further away. The separate scoring is blind to that interaction.

So the best passage can rank low.

Put those two facts together. Approximate (built for speed) plus separate (each passage scored alone) means the ranking is a rough guess, not a verdict. The truly best passage is reliably somewhere in the top fifty, but it is not reliably at position one. Often it sits at rank eight, or fifteen, or twenty, buried under passages that merely looked similar.

For a question with one clean answer near the top, that is fine. For a hard question where the real evidence is a few slots down, handing the model only the top few passages means the model never sees the passage it most needed. The answer suffers, and it is not the model's fault: it was never shown the right source.

The fix is not to make the first pass slower. It is to take a second, closer look at just the top handful of candidates the first pass already found. Cheap-and-wide first, careful-and-narrow second. The next lesson builds that second look.

Key takeaways