Timeouts and deadlines

A clear "no" beats waiting forever.

You ring a shop to ask if they have something in stock. If they say no, fine, you call the next shop. But if the line just rings and rings and nobody picks up and nobody hangs up, you are stuck. You cannot try the next shop because you are still holding this dead line. The worst outcome is not a "no." It is no answer at all.

A call from the gateway to a provider can hang exactly like that. The provider stalls and never closes the connection, so your request sits there waiting forever, holding resources, never failing and never succeeding. Without a rule that says "I have waited long enough, give up on this one," a single hung provider can freeze your whole request.

Two rules prevent that, and they work at different scopes. A timeout bounds one attempt. A deadline bounds the whole operation across every retry. The gatekeeper monk does not wait at a silent door indefinitely, and he does not let the total wait run past what the traveler can bear.

One bounds an attempt; the other bounds the budget.

A timeout is the longest you will wait for a single attempt before declaring it dead. "Wait five seconds for an answer; if none comes, give up on this try." It is a stopwatch on one call. Without it, a hung provider hangs you.

A deadline is the total time budget for the whole operation, including every retry and every backoff wait in between. "The caller can wait twelve seconds, full stop." It is a stopwatch on the entire effort, not on any one call.

They have to work together, because retries with backoff can quietly blow past what the caller will tolerate. Say each attempt gets a five-second timeout and the whole operation gets a twelve-second deadline. After one failed attempt (5s) plus a short backoff wait, you have only enough budget left for roughly one more full attempt. So that pairing gives you about two attempts before the deadline cuts you off, no matter how many retries your backoff rule would have allowed.

The deadline is the boss. It does not care how the time was spent (a slow attempt, a backoff wait, a retry); once the budget is gone, the operation fails and the caller gets a clean, prompt error instead of an open-ended hang.

Setting the timeout is a tradeoff.

Pick the timeout too short and you cut off slow-but-valid responses. Some real answers genuinely take a while, especially long generations, and a trigger-happy timeout throws away work that was about to succeed. Pick it too long and you waste that whole stretch sitting on calls that were never coming back, which is exactly the hang you were trying to avoid.

The way out is to set the timeout from the latency you actually see, not a guess. If your p99 latency is four seconds (meaning ninety-nine percent of real responses arrive within four seconds), a two-second timeout would cut off that slow but legitimate tail. A timeout a bit above the p99 keeps the genuine slow responses and still kills the truly stuck ones.

Drag the timeout line left and right. Pull it in tight and you "serve" fast but cut off the slow tail of real responses. Push it out and you keep more of them but wait longer on the dead ones. There is no free setting; there is only the tradeoff you choose on purpose.

Key takeaways