Lesson 6.1
The Latency Budget
Four hops in sequence means four places to spend time. You get one number to defend, the p99, and it is the sum of the stages.
A relay race on a fixed clock.
Picture a relay with four runners. Runner one hands the baton to runner two, who hands it to three, then four. They do not run at the same time. They run one after another, so the team's time is the four leg times added up. Slow down any one runner and the whole team is slower by exactly that much.
A RAG request is that relay. It embeds the question, searches for passages, reranks them, then generates the answer. Each step waits for the one before it. So the time the user waits is the sum of the four steps, and a budget is just the clock you promised to beat.
The typical time, and the slow one.
Run the same request a thousand times and the answers do not all take the same number of milliseconds. Some are quick, a few are slow. To talk about speed honestly you pick a point in that spread. Two points matter.
The first is the middle, the time half your requests beat (the median, written p50). That is the typical experience. The second is the slow edge, the time only one request in a hundred is worse than (the 99th percentile, written p99). That is the tail.
p50 the time half your requests are faster than. The day-to-day feel.
p99 the time only 1 in 100 requests is slower than. The worst your users routinely hit.
The tail is what people actually feel. A user who fires off a hundred questions in a session is almost certain to land on that slow one, and one bad wait colors the whole impression. So the number you defend is the p99, not the p50. Promising "fast on average" hides the request that makes someone give up.
Drag a stage and watch the budget blow.
Below is the relay drawn as one bar, split into the four stages. Grab the handle under any stage and drag it slower or faster. The total is the sum of all four. There is a budget line at 800ms. Make one stage greedy and watch the total cross the line, even when the other three are lean.
Drag a stage's handle to change its cost
Stage times are illustrative, not measured. They show how four sequential hops add into one number.
Two lessons fall out of dragging. First, because the stages add up, the slowest one dominates: it is the easiest place to lose your budget and the most valuable place to fix. Second, you cannot spend the same millisecond twice. Every millisecond you hand to reranking is a millisecond generation has to do without, if the total is going to fit.
A worked sum.
Say your budget is a p99 of 800ms. Suppose, for illustration, the stages come in like this: embedding the question takes 90ms, searching the index takes 120ms, generating the answer takes 520ms. That adds to 730ms, comfortably under 800. You have 70ms of headroom.
Now you want to add reranking, which lesson 5.3 showed costs roughly 150ms for a sharper ordering. 730 plus 150 is 880ms. You are 80ms over. The reranking is not free and your headroom does not cover it, so something has to give: a smaller candidate set, a faster model, a cheaper stage elsewhere, or a looser budget. The arithmetic decides, not the wish.
Sequential stages add. The p99 you defend is the sum of the stage p99s, so the slowest hop sets the pace and every new feature has to fit inside the leftover milliseconds, not on top of them.
What a budget is not
An average
A mean hides the tail. A handful of very slow requests barely move the average but ruin the p99, which is the one users remember.
A per-stage target
The budget is on the whole request. A stage can be fast and the request still slow, because the total is what the user waits through.
A nice-to-have
It is a ceiling you hold the p99 under. Without one, every new step quietly adds time and nobody notices until the request feels sluggish.
Key takeaways
The p50 is the typical time, the p99 is the slow tail, and the tail is what users feel, so the p99 is the number you defend.
The hops run in sequence, so the total is their sum and the slowest hop dominates the budget.
A budget is a ceiling you hold the p99 under, and every new feature has to fit inside the leftover milliseconds.