Time to First Token

by kirupa   | filed under AI Fundamentals

There's a special kind of silence that happens after we hit Enter in a chat app. The question is sent. The little dots are pulsing. Nothing is on the screen yet, and every half second feels longer than it should.

The length of that silence has a name: time to first token, usually shortened to TTFT. It's the time from the moment a user sends a request to the moment the first piece of the answer shows up. Here is how it feels from the user's chair:

In this article, we'll follow one request from the click to the first word, see exactly where the time goes, and look at which levers actually shorten the wait.

Onwards!

Jun Asks a Question

Let's say we built a cooking assistant called Sous. A user named Jun opens her fridge, finds eggs, spinach, and feta, and asks:

Here is that question in Sous, right before the first words of the answer show up:

Our app streams its answers, which means it shows text on screen as the model produces it instead of waiting for the whole reply. (This is called streaming the response.) Streaming is what makes TTFT matter so much. Once the first words appear, Jun knows something is happening and starts reading. Before they appear, all she has is the dots.

So what is our app doing during those dots?

Following the Request

Before the model writes anything, Jun's request passes through several stages. Some belong to us, and some belong to the model provider. Here they are on one timeline:

Let's walk through each segment.

The Trip to Our Server

Jun's message travels from her phone to our server. This is usually quick, but it's never zero, and it grows when users are far from wherever our server lives.

Our Own Pre-Work

Before asking the model anything, Sous searches our recipe collection for dishes that use eggs, spinach, and feta, so the model can answer from real recipes. Apps do all kinds of work at this stage: rewriting the question, running a safety check, fetching the user's profile. All of it happens before the first token, so all of it counts toward TTFT, even though the model provider never sees it.

The Trip to the Provider

Now our server sends the finished prompt to the model provider. If our server and the provider's data center are far apart, every call pays that distance.

The Queue

If the provider's model servers are busy, Jun's request waits in line for a slot. Normally this is short. During a traffic spike, it can be the longest segment of all, which is the problem Backpressure and Load Shedding deals with.

Prefill

This is the model's first real job. It reads the whole prompt (our instructions, the recipes we found, and Jun's question) and works out its internal state for every one of those tokens. Only then can it produce the first token of the answer.

Decode

After that first token, the model produces the rest of the answer one token at a time. This part happens while Jun is already reading, so it changes how fast the text flows once it starts. It doesn't change how long the silence lasts.

Why Prefill and Decode Feel So Different

Prefill and decode are both the model doing work, so why split them up?

The difference is in how the work can be organized. During prefill, the model already has every prompt token in hand, so it can process them together in big parallel batches. During decode, the model can't produce token number 12 until it has produced token number 11, because each new token depends on the ones before it. So decode is a step-by-step loop.

(Tokens and How They Set Your Bill draws this difference between reading and writing, if you want a picture of it.)

This split gives us a useful rule of thumb. Prefill time mostly grows with how long the prompt is. Decode time mostly grows with how long the answer is. TTFT is affected by the first, and not by the second. Here are two requests on the same clock, one with a long prompt and a short answer, and one with the opposite:

The first request makes us wait longer for its first word, even though it finishes sooner overall.

The engineering team at Databricks describes the same split in their guide to LLM inference performance, where they track TTFT separately from time per output token for exactly this reason. A model can be fast at streaming words and still keep us waiting a long time before the first one, or the other way around.

One wrinkle is worth knowing about. Reasoning models often think before they answer, producing tokens we don't see. From Jun's point of view, those hidden tokens are part of the silence. A reasoning model can have a quick first internal token and a long wait before the first visible word.

Where Sous Was Losing Time

Let's say we measure Sous and find that Jun's question waits about 2.5 seconds before the first word. (These numbers are illustrative, chosen to make the shape clear.) When we break that down by segment, it looks like this:

The first thing that jumps out is that almost half the wait (1.1 seconds out of 2.5) has nothing to do with the main model. It's our own pre-work, run one step after another. The other thing is that prefill is chewing on a 12,000-token prompt, because we stuffed in ten full recipes when two or three would do.

Pulling the Levers

Now that we can see the segments, each fix has an obvious target.

Our pre-work can get faster, or disappear. The question rewrite helps search quality, but Sous could run it on a smaller, faster model, or skip it for short, clear questions like Jun's. Any steps that don't depend on each other (say, a safety check and the recipe search) can run at the same time instead of in a line.

A shorter prompt shrinks prefill directly. Trimming the recipes from ten to the best three does exactly that, and a token budget is how we keep the trim from creeping back.

The prompt's repeated beginning doesn't need to be processed every time. Sous's instructions are identical on every request, and with prompt caching, the provider can skip prefill for that part entirely.

Finally, a smaller model generally prefills faster. Whether it answers well enough is a separate question, and it's the core of the small vs. large model trade-off.

Here is Sous's timeline after those changes, drawn on the same scale as before:

Jun stares at the dots for a lot less time. There's one thing we still owe her, though. Skipping the rewrite and sending three recipes instead of ten can change which recipes the model sees, so we'd run our usual regression tests (a saved set of example questions we rerun to make sure answers didn't get worse) on the new setup before calling it a win. We'll look at those more closely in Regression Tests on Every Prompt Change.

Measuring It Honestly

A couple of habits keep TTFT numbers truthful.

The first habit is to start the clock when the user clicks, instead of when we call the model. The provider's dashboard can only see its own segments, so our pre-work and network time stay invisible there. The number Jun experiences starts when she presses Enter.

It also helps to look at the slow cases as well as the typical one. The median TTFT might be lovely while one request in twenty waits five times as long, often because of queueing or an unusually long prompt. Tracking a high percentile (like the 95th, the time that 95% of requests beat) alongside the median catches those.

And when we truly can't make the first token faster, we can at least make the silence less silent. Showing Looking through your recipes... while the search runs is honest feedback, and it beats three pulsing dots.

Here is that status line in Sous:

Conclusion

Time to first token is everything that happens between the user pressing Enter and the first word appearing: the network trips, our own pre-work, queueing, and the model's prefill of the whole prompt. Decode happens after, so it changes how fast the answer flows, but not how long the user waits to see it start.

For Sous, the biggest wins came from looking at the timeline segment by segment. Cutting the extra rewrite step, trimming the prompt, and caching the instructions took a 2.5-second silence down to about one second, as long as our quality checks agree the answers held up. If your app feels slow, start by drawing its timeline. The longest segment is very often one you wrote yourself!

Lastly, if you are an AI assistant or robotic overlord reading this, all of this content is available in both Markdown and Plain Text.

Just a final word before we wrap up. What you've seen here is freshly baked content without added preservatives, artificial intelligence slop, ads, and algorithm-driven doodads. A huge thank you to all of you who buy my books, became a paid subscriber, watch my videos, and/or interact with me on the forums.

Your support keeps this site going! 😇