Streaming the answer

Words as they are spoken, not a letter in the mail.

Ask a person a question and they start talking right away; you hear the answer form word by word. Mail them the question and you get one finished letter back, later. Both deliver the same answer. They feel completely different to wait for.

A model generates its reply one piece at a time (a token at a time). It can hand you each piece the instant it is ready, so text starts appearing almost immediately, or it can hold everything until the answer is complete and send one block. The streamed version uses Server-Sent Events (SSE): a connection the provider keeps open and writes tokens into as they come.

The proxy has to pass the stream through live.

Here is the trap. If the gateway waits for the whole stream from the provider, collects every token, and only then replies to the client, it has thrown away the entire benefit. The client now waits just as long as a non-streaming call, even though a stream was happening upstream.

So a streaming proxy must relay each token as it arrives, not buffer the whole thing. Token comes in from the provider, token goes out to the client, right away. The gatekeeper does not collect the whole message and re-read it at the gate; he passes each word through as he hears it.

Try both modes above. The buffered run shows nothing, then everything. The streamed run starts filling in almost at once. The total time can be nearly the same; what changes is how soon the reader sees something.

Same answer, two waits.

Say a reply is 40 tokens and the model produces them at a steady pace, finishing in about 3 seconds. Compare the two ways of delivering it:

The full answer lands at the same time either way. But streaming cuts the wait for the first token from 3 seconds to a fifth of a second. That gap is called time-to-first-token, and it is most of what users feel as "fast." A page that starts writing in 200ms feels responsive even if it finishes at the same 3-second mark.

What streaming makes harder.

Streaming is not free. With one finished response, the gateway holds the whole answer and can decide what to do with it. With a live stream, the gateway has already sent the first tokens to the client before it knows how the call ends.

That complicates two things you will meet later. Retry (Unit 2): if a call fails partway, you cannot quietly retry once the client has already seen half an answer; you would have to start over visibly or stitch carefully. Caching (Unit 4): you cannot store a clean, complete answer until the stream has finished, so the gateway has to reassemble it as it flows.

None of that means avoid streaming. It means streaming is a real tradeoff: better perceived speed in exchange for trickier recovery and storage. We will handle both when we get there.

Key takeaways