Lesson 6.3
Does Rerank Earn Its Cost?
Reranking buys recall and spends latency. Whether that trade is worth it depends on your budget, and you decide it with numbers, not vibes.
A trade, not an upgrade.
Reranking is tempting because it almost always makes the ordering better. The cross-encoder from unit 5 reads the query and each passage together and sorts them more carefully than first-pass search did. More of the truly relevant passages end up near the top (higher recall in your top results).
But it is not a free upgrade, it is a trade. You buy better recall and you pay in milliseconds, and those milliseconds come straight out of the p99 budget from lesson 6.1. So the real question is not "is reranking good?" It is "does the recall it buys justify the latency it spends, given my budget?" That is a numbers question.
The three numbers that decide it.
Three quantities settle the call. The recall you gain by reranking (how many more of the right passages reach the top). The milliseconds you add (the cost of the cross-encoder pass). And the headroom in your budget (how much of the p99 ceiling is still unspent). Reranking is worth it when the recall gain matters for your use and the added milliseconds fit under the ceiling.
When it pays
The answer has to be right (legal, medical, support where a wrong passage is costly), and the budget has room to absorb the extra time. The recall gain is worth more than the milliseconds.
When it does not
Latency is already tight, or first-pass recall is already high enough that the reorder barely changes the answer. You would spend real milliseconds to gain almost nothing the user notices.
Make the call.
Decide it with the numbers
You run a support assistant. Your p99 budget is 800ms. Without reranking, the pipeline lands at a p99 of 700ms and recall@5 is 0.78 (78 percent of the right passages reach the top five). Turning reranking on adds 150ms and lifts recall@5 to 0.91. The numbers are illustrative.
Your task:
- Add the 150ms to the 700ms p99. Does the result still fit under 800ms? By how much is it over or under?
- Decide: do you turn reranking on? Justify it in one sentence using the recall gain and the budget.
- If the answer is "over budget," name one thing you could cut elsewhere to make room, and roughly how many milliseconds you would need to find.
A hint, not the answer: 700 plus 150 is 850, and the ceiling is 800. So "just turn it on" breaks the budget by 50ms. The honest move is to decide whether the 0.13 recall gain is worth finding those 50ms somewhere, or worth a slower promise.
Measured against a baseline.
Notice what made the exercise answerable: you had numbers. A recall figure, a latency figure, a ceiling. The honest verdict on any upgrade comes from comparing it to a baseline, the same pipeline with the feature turned off, measured the same way. "Reranking feels better" is not a verdict. "Reranking lifts recall@5 from 0.78 to 0.91 and costs 150ms against an 800ms budget" is.
This is exactly the headline of monkrag's second stage: a reproducible measurement that says, in numbers, whether each upgrade earns its cost against a naive baseline. The point of the system is not that it has reranking. It is that it can prove, on its own benchmark, whether reranking was worth it.
Reranking is a trade: recall bought with latency. You decide it with three numbers (recall gained, milliseconds added, budget headroom) measured against a baseline with the feature off, not with a feeling.
Key takeaways
Reranking buys recall and spends latency, so it is a trade decided against the p99 budget, not a free upgrade.
It pays when the answer must be right and the budget has room; it does not when latency is tight or recall is already enough.
The honest verdict is measured against a baseline with the feature off, in numbers, which is the headline of monkrag.