Lesson 4.2
Streaming the Answer
The model writes one word at a time. Streaming shows each word as it arrives, so the answer feels fast even when the whole thing takes a while.
Watching a kettle versus watching it fill a cup.
Two ways to serve tea. In the first, you wait, staring at a closed lid, until the whole pot is ready, then it arrives all at once. In the second, the cup fills in front of you a trickle at a time. The total wait is the same, but the second one feels far shorter, because something is happening the whole time.
That is the difference between waiting for a full answer and streaming it. The model does not think up the whole reply and hand it over in one piece. It writes one small chunk at a time, each chunk a word or a piece of a word (the precise term is a token). Streaming means we show each token to the reader the instant it is produced, instead of holding them all back until the last one lands.
Streaming does not make the model faster. It changes when the reader first sees something, which is most of what "fast" feels like.
One token after another.
A language model generates left to right, one token at a time, and each new token depends on all the ones before it. It cannot skip ahead or write the end first. So a fifty-token answer is fifty small steps in a row, each taking a few milliseconds.
This is why two clocks matter, not one. The first is how long until the very first token shows up (the precise term is time to first token). The second is how long until the whole answer is done. Streaming attacks the first clock. The reader starts reading the moment word one appears, while the model is still busy writing word twenty.
A worked timeline.
Say a grounded answer takes 1.2 seconds to write in full, and the first token is ready at 0.3 seconds. (These are illustrative numbers, not a benchmark.) Here is how the two approaches land for the reader.
Wait for the full answer
0.0s question sent, blank screen
0.3s first token ready, still blank screen
1.2s the whole answer appears at once
The reader stares at nothing for 1.2 seconds.
Stream the answer
0.0s question sent, blank screen
0.3s first word appears, the reader starts reading
1.2s last word lands, but the reader has been busy since 0.3s
The felt wait is 0.3 seconds, not 1.2.
Total work is identical in both: the model still spends 1.2 seconds writing fifty tokens. The only thing that moved is the reader's experience. The wait they actually notice (the precise term is perceived latency) dropped from 1.2 seconds to 0.3.
Streaming trades a long silent wait for a short one followed by a steady trickle. The clock on the wall is unchanged; the clock in the reader's head is four times shorter.
Where this fits the budget.
Unit 6 puts the whole pipeline under one time budget and measures the worst-case wait across the four hops. Streaming does not buy you a single millisecond on that measured total, so it will not move the p99 number. What it buys is the gap between the number a stopwatch records and the speed a person feels. Both are real; they are just different clocks, and a serving system has to mind both.
Key takeaways
A model writes one token at a time, left to right; streaming shows each token the instant it is produced instead of holding them all back.
Two clocks matter: time to first token (when the reader first sees anything) and total time. Streaming shrinks the first, not the second.
Total work is unchanged, so the measured latency budget (Unit 6) is unmoved. Streaming improves only the wait a person feels, the perceived latency.