Scout and Judge

A scout and a judge.

Think of a hiring funnel. A scout skims a thousand resumes fast and flags fifty worth a look. Then a judge sits down with the question and each of those fifty resumes side by side, reads them properly, and ranks the best handful. The scout is cheap and broad. The judge is slow and sharp. You would never run the judge over a thousand resumes, and you would never trust the scout's order for the final call.

Retrieval works the same way. The scout is the bi-encoder from the last lesson. The judge is a different kind of model called a cross-encoder. Together they are the scout-then-judge pattern, and almost every serious retrieval system uses it.

The scout: separate, precomputed, cheap.

The bi-encoder (the scout) turns the question into a vector and turns each passage into its own vector, separately. The passage vectors are computed once, ahead of time, and stored, so at question time you only encode the question and measure which stored vectors sit closest. Because the heavy work happened earlier, the scout can run over the entire collection, millions of passages, in a blink.

The price of that speed, as we saw, is that the scout never looks at the question and a passage at the same time. It compares two summaries. Good enough to find a shortlist, too blunt to perfectly order it.

The judge: together, live, expensive.

The cross-encoder (the judge) does the thing the scout cannot. It feeds the question and one passage through the model together, as a single joined input, and reads them in light of each other. Now the model can notice that this exact passage answers this exact question, not just that their summaries land nearby. That joined reading is far more accurate.

It is also far more expensive, and for a clear reason: there is nothing to precompute. The judge's score depends on the question, so it cannot be calculated in advance. You have to run the full model once for every single passage you want scored. Score fifty passages, run the model fifty times. That is why the judge only ever sees a small shortlist, never the whole collection.

Drag the judge's depth.

The scout hands over fifty candidates. The one dial you control is how many of them the judge actually re-reads, the rerank depth. Drag the handle and watch two numbers move: how good the final ranking gets (precision), and how much extra time the judging adds.

Notice the shape. Precision jumps as you go from judging nothing to judging the top ten, then the gains taper off: by the time you are judging forty or fifty, you are paying for passages the scout had already ranked low for good reason. Latency, meanwhile, just keeps climbing in a straight line, because each extra passage means one more full run of the judge. More judged means better ranking but more time, and the curve bends while the cost does not.

Key takeaways