# Streaming Responses

by [kirupa](https://www.kirupa.com/me/index.htm) | filed under [AI Fundamentals](https://www.kirupa.com/ai_fundamentals/index.htm)

Picture two restaurants. In the first one, the kitchen cooks your whole meal and brings everything out at once, after 20 minutes. In the second one, the bread shows up after one minute, the salad a few minutes later, and so on. The total time is the same. The second restaurant feels a lot faster, though, because you're eating almost right away.

Language models can work like either restaurant. By default, an API call waits until the model has written its entire answer, then sends it all back at once. With **streaming**, the model's answer arrives a few words at a time, while it's still being written.

In the following sections, we'll look at why streaming feels so much faster without actually being faster, what the stream looks like on the wire, and the handful of surprises that come with it.

Onwards!

## The Spinner Problem

Let's say we've added an ***Explain this error*** button to a code editor. You click it, and the model writes about 400 words explaining what went wrong and how to fix it.

A model doesn't produce its answer all at once. It writes one token at a time (a token is a word or a piece of a word), and each token takes a little time to generate. To keep the math simple, let's say the whole 400-word answer takes about 10 seconds to write. That number is made up, and real speeds vary a lot by model and load, but the shape is what matters. Here is what the user sees with and without streaming:

![Two timelines over ten seconds. With no stream, the user waits the full ten seconds and then gets the answer. With streaming, words start appearing after about 0.6 seconds and the user reads as they arrive. A handwritten note points out the same finish time.](https://www.kirupa.com/ai_fundamentals/images/streaming_responses_timeline.png)

Without streaming, the user stares at a spinner for the full 10 seconds. With streaming, words start showing up in well under a second, and the user starts reading while the rest is still being written. People usually read slower than models write, so by the time they reach the end of the first paragraph, more has already arrived.

The answer finishes at the same moment either way. What changed is the wait before anything shows up. That first wait is called **[time to first token](https://www.kirupa.com/ai_fundamentals/time_to_first_token.htm)**, and it's often the number that decides whether an AI feature feels fast or sluggish.

## What the Stream Looks Like

So how does a stream actually travel? Most model APIs use a simple, old web standard called **Server-Sent Events** (aka SSE). We ask for a stream (usually by setting `stream: true`), and instead of one big response body, the server keeps the connection open and sends a series of small text events:

![The provider sends the answer to our app as a series of small events, each carrying a few characters of text such as Your, code, and calls, over a response with the content type text/event-stream.](https://www.kirupa.com/ai_fundamentals/images/streaming_responses_events.png)

If we peeked at the raw response, it would look roughly like this (the exact fields differ by provider):

```text
data: {"delta": "Your"}

data: {"delta": " code"}

data: {"delta": " calls"}

data: {"delta": " map()"}

...

data: [DONE]
```

Each `data:` line carries a small piece of new text, often called a **delta**. Some APIs end with a marker like `[DONE]`, and others send a named ***message stopped*** event. Our job on the receiving end is simple: keep appending each delta to what we already have, and redraw.

## Putting It on the Screen

Here is what that appending looks like in the browser, at three moments:

![The same answer bubble at three moments. After 0.6 seconds it shows Your code calls with a cursor. After 2 seconds it shows three lines. After 10 seconds the full answer is there, the cursor is gone, and the bubble turns green.](https://www.kirupa.com/ai_fundamentals/images/streaming_responses_growing_bubble.png)

And here is a small version of the code that does it, using the browser's `fetch` and a stream reader:

```js
const response = await fetch("/api/explain", {
  method: "POST",
  body: JSON.stringify({ error: errorText }),
});

const reader = response.body.pipeThrough(new TextDecoderStream()).getReader();
let answer = "";

while (true) {
  const { value, done } = await reader.read();
  if (done) break;
  answer += extractDeltas(value);   // parse the "data:" lines
  render(answer);
}
```

The `extractDeltas` helper is where we parse those `data:` lines. In real code, it also has to handle an event that gets split across two chunks. Most provider SDKs do this parsing for us, so we usually just loop over the events they hand back.

## Our Server Is in the Middle

In a real app, the browser usually doesn't talk to the model provider directly, because that would put our API key in the browser. Our own server sits in between, so the stream has to make it through two hops:

![Events flow from the provider to our server and then on to the browser. A buffer inside our server is marked with a handwritten note: don't hold them here!](https://www.kirupa.com/ai_fundamentals/images/streaming_responses_two_hops.png)

The thing to watch for here is buffering. If our server, a proxy, or a load balancer collects the whole response before passing it along, we're right back to the spinner, even though every piece of our code thinks it's streaming. Compression middleware and some reverse proxies like to do this by default, so if a stream works locally but not in production, buffering is the first place to look.

## The Surprises

Streaming changes a few things that we took for granted with ordinary requests.

### Errors Arrive Mid-Answer

With a normal request, a failure comes back as an HTTP error status, like a 500. With a stream, we've already sent a `200 OK` and half an answer before the model runs into trouble, and there's no way to change the status code after that. So errors have to arrive as events inside the stream, and our UI needs a way to show ***the answer stopped partway through*** next to the text that's already on screen.

Here is what that can look like in the editor's help panel:

![The Explain this error panel in a code editor. A note shows the Explain this error button was clicked. Explain this error replies, Your code calls. Below the half-written answer, a note reads: Answer stopped partway.](https://www.kirupa.com/ai_fundamentals/images/streaming_responses_chat_stopped.png)

### JSON Isn't Done Until It's Done

If we asked the model for JSON, the stream gives us broken JSON until the very last event arrives. `{"title": "Undefined va` isn't something `JSON.parse` can work with. We can wait for the whole thing, or use a parser built for partial JSON, but we can't treat each chunk as a finished object.

### Cancel When the User Leaves

If the user closes the panel or clicks Stop, we should close our connection to the provider too, so we stop paying for tokens nobody will read. It's easy to forget, because nothing visibly breaks when we don't do it.

### Usage Numbers Come Last

Token counts for the response usually arrive only at the very end of the stream (and some APIs only send them if we ask). If we log costs, we need to grab those numbers from the final event.

## When Not to Stream

Streaming is for moments when a person is watching words appear. If nobody's watching, it mostly adds complexity. A nightly job that tags 10,000 support tickets, an agent's internal tool-calling steps, or a classifier that returns one word gain almost nothing from it. Those are better served by ordinary requests or by [async processing with queues](https://www.kirupa.com/ai_fundamentals/async_processing_with_queues.htm).

## Conclusion

We started with a 10-second answer and a user staring at a spinner. Streaming didn't make the model any faster. What it did was send each piece of the answer as soon as it existed, so the first words showed up in well under a second and the user could start reading right away.

Under the hood, that's a long-lived connection carrying small events, with our code appending each delta to the screen. It comes with some homework: keep every hop from buffering, report errors inside the stream, wait for complete JSON, cancel upstream when the user leaves, and pick up the usage numbers at the end. For any feature where a person is watching the answer appear, that homework is almost always worth it.