Lesson 5.3
Retrieve, Then Rerank
The two-stage pattern: cast a wide net cheaply, then rerank the top of it carefully. More quality, more latency. The question is how much of each.
Wide net, then a close look.
A fisherman casts a wide net to haul in a big, messy catch, then sorts the keepers on deck by hand. The net is fast and indiscriminate. The sorting is slow and careful, but it only touches what the net already brought up. That is the whole pattern: retrieve wide and cheap, then rerank narrow and careful.
We have the two players from the last lesson, the scout (bi-encoder) and the judge (cross-encoder). This lesson is about wiring them into the pipeline you have been building, and about the one number that now matters: how much you rerank.
The change to the pipeline.
Until now the flow was: search returns the top few passages, those go into the prompt, the model answers. Reranking splits that first step in two. Search now returns a wide list, say the top fifty (call it N). The judge re-scores those fifty and keeps only the best few, say five (call it k). Those five are the ones that go into the prompt.
before
question → search returns top 5 → prompt → answer
after
question → search returns wide N=50 → rerank to best k=5 → prompt → answer
The prompt still gets five passages, same as before. What changed is how those five were chosen: not the scout's rough order, but the judge's careful one over a much bigger candidate pool. The best passage, which the scout might have left at rank twenty, now has a chance to make the final five.
The dial: how many you rerank.
N is the dial. It is how deep into the scout's pile the judge reaches, the rerank depth you dragged in the 5.2 demo. Turn it up and the final five are chosen from a wider, better-sorted pool, so quality climbs. But every extra passage in N is one more fresh run of the judge, so latency climbs too. Quality up, latency up, both driven by the same knob.
And recall the shape from the demo. Quality rises with diminishing returns: the jump from N=10 to N=20 buys a lot, the jump from N=40 to N=50 buys almost nothing. Latency, though, rises in a roughly straight line, because the judge's cost is simply (cost per passage) times N. So the two curves diverge. Past a point you are paying steady extra milliseconds for quality you can barely measure.
Reranking adds one tunable number, the rerank depth N. Bigger N means better final passages but more latency. Because quality flattens while latency keeps climbing, the right N is a budget question, not a "more is better" one.
Which sets up the real question.
So reranking is clearly worth something: it rescues the good passage the scout buried. The honest question is not "does it help" but "is the help worth the milliseconds it costs, given everything else competing for the same time budget?" Embedding, search, and generation each want a slice of the clock too. Reranking has to earn its slice.
That is exactly the question the next unit takes head on: a fixed latency budget split across all the stages, and a measured verdict on whether reranking pays for itself. We have the dial; next we learn to spend it.
Key takeaways
Reranking splits the first step in two: search returns a wide N (say 50), then a reranker keeps the best k (say 5) for the prompt.
The rerank depth N is the dial: bigger N means better final passages (with diminishing returns) but steadily more latency.
Because quality flattens while latency climbs, choosing N is a budget decision, which sets up the latency-and-cost question in the next unit.