# Context Window Limits

by [kirupa](https://www.kirupa.com/me/index.htm) | filed under [AI Fundamentals](https://www.kirupa.com/ai_fundamentals/index.htm)

In the [previous article](https://www.kirupa.com/ai_fundamentals/tokens_and_your_bill.htm), we saw that every turn of a chat sends more tokens than the last. That happens because a model doesn't remember our conversation. Not even the message we sent ten seconds ago. Every time a customer sends a message to **Biscuit**, the help desk bot for the Crumb & Co. bakery, our code packs up the entire conversation so far, sends all of it to the model, and the model reads the whole thing from the top as if it were seeing it for the first time.

That works fine until the pile gets too big. Every model has a hard limit on how much text it can look at in one go, and that limit is called the **context window**. In the following sections, we'll look at what goes into that window, what happens when a conversation outgrows it, how to make room, and why ***it fits*** isn't the same as ***the model will use it well***.

Onwards!

## The Model's Desk

A good way to picture a context window is as a desk with a fixed amount of surface. Anything the model is going to consider has to be laid out on that desk at the same time. If something isn't on the desk, the model doesn't know it exists.

What goes on the desk? A lot more than the last thing the customer typed. Here is what every one of Biscuit's requests has to fit inside the window:

![A context window drawn as one fixed tray holding four blocks: the system prompt, the bakery facts, the conversation so far, and an empty dashed block reserved for the model's reply. A handwritten note points at the empty block: the reply needs room too.](https://www.kirupa.com/ai_fundamentals/images/context_window_limits_desk.png)

The last block is the one people forget. The window is measured in tokens, and it covers both what we send in **and** what the model writes back. If our input uses nearly all of the window, the model has almost no room left to answer. Most APIs also let us cap the reply with a separate maximum output setting, and our input plus that cap still has to fit inside the window.

How big is the desk? It depends on the model. At the time of writing, many models accept a few hundred thousand tokens, and some accept a million or more. Those numbers sound endless, and for a single question they are. They stop feeling endless once a conversation, a pile of documents, and a stack of tool results all have to share the same space.

## Watching the Desk Fill Up

Let's watch it happen with Biscuit. From [Tokens and How They Set Your Bill](https://www.kirupa.com/ai_fundamentals/tokens_and_your_bill.htm), every request sends 1,500 tokens of system prompt and bakery facts, a typical customer question is about 25 tokens, and a typical reply is about 100 tokens. Since every earlier question and reply gets sent again, the conversation part of the request grows by about 125 tokens per turn.

Real context windows are far bigger than Biscuit's requests, so to see the wall, we'll pretend our model has a tiny, made-up window of 3,000 tokens. Here is what the request looks like at a few points in a long chat, including the 100 tokens we leave free for the reply:

![Five bars show the tokens in Biscuit's request at turns 1, 5, 10, 15, and 20. Each bar has the same 1,500 tokens sent every time, a conversation band that grows each turn, and 100 tokens of room for the reply. With 3,000-token window, turns 1, 5, and 10 fit, while turns 15 and 20 run past the limit and are outlined in red.](https://www.kirupa.com/ai_fundamentals/images/context_window_limits_growth.png)

Turn 12 is the last one that fits. Its input is 2,900 tokens, and with 100 tokens of room for the reply, it lands exactly on the 3,000 limit. On turn 13, the input alone is 3,025 tokens, so it can't fit no matter how short the reply is.

A real window would take many more turns to fill, but the shape of the problem is identical. The fixed part never shrinks, and the conversation part only grows. If a customer pastes in a long order history or a big chunk of an email thread, that text rides along on every turn after it too, and the wall arrives much sooner.

## What Happens at the Wall

So what happens on turn 13, when the request is bigger than the window?

The model doesn't skim, and it doesn't pick out the important parts on its own. Most APIs simply reject the request with an error saying the input is too long. That's the good outcome, because at least we know something went wrong.

The trouble starts when an app or framework avoids the error by cutting text before sending it, without much care for what gets cut. Imagine that on turn 2, a customer named Priya told Biscuit she's allergic to walnuts. If the cutting removes the oldest turns first, that message is the first thing to go. On turn 14, she asks which cake she should order for her birthday, and Biscuit happily recommends the banana walnut loaf.

Here is what that looks like from Priya's side of the chat:

![The Biscuit chat window on the Crumb & Co. bakery website. Priya's early message, I'm allergic to walnuts, is drawn faded with a dashed outline. A note below it reads: Oldest turns trimmed to fit. Priya asks, Which cake should I order for my birthday? Biscuit replies, You'll love our banana walnut loaf! A handwritten note flags that reply: She's allergic!](https://www.kirupa.com/ai_fundamentals/images/context_window_limits_chat_walnuts.png)

The model has no idea anything was removed. It answers with what's left on the desk, and there's no error anywhere. This is the failure to watch for: the bot suddenly ***forgets*** a detail from earlier or stops following a rule, and nothing in our logs says why.

## Making Room

Since something has to leave the desk eventually, the job is to decide **what** leaves instead of letting it fall off at random. There are three common ways to do this, and real apps often combine them:

![Three panels show the same window. Trim crosses out turns 1 to 8 and keeps turns 9 and 10. Summarize replaces turns 1 to 8 with one short summary note. Retrieve replaces the full bakery facts with only the gluten-free facts the question needs, while the rest of the facts stay on the shelf.](https://www.kirupa.com/ai_fundamentals/images/context_window_limits_make_room.png)

**Trimming** keeps the newest turns and drops the oldest ones. It's simple and predictable. The catch is that whatever was in the dropped turns is gone, so we always pin the system prompt, and anything else that must survive, so it never gets trimmed.

**Summarizing** replaces older turns with a short summary, usually written by a model. Priya's first eight turns might become one short note saying she's allergic to walnuts, wants something for a birthday, and prefers chocolate. We lose the exact wording, but we keep the facts that matter, as long as the summary is good. Since a summary is only as good as whatever wrote it, this is worth testing with real conversations.

**Retrieving** stops putting all of the bakery facts on the desk every time. When Priya asks about gluten-free bread, we look up only the gluten-free section and send that. This is the idea behind retrieval-augmented generation (aka RAG), and it's how apps work with document collections far larger than any context window.

## Fitting Isn't the Same as Using

Since windows keep getting bigger, you might wonder whether we can just buy a bigger desk and skip all of this. As it turns out, no.

Researchers have found that models don't pay equal attention to everything on the desk. In the 2023 paper [Lost in the Middle](https://arxiv.org/abs/2307.03172), Nelson Liu and colleagues placed the one passage containing an answer at different positions in a long input. The models did best when that passage was near the beginning or the end, and noticeably worse when it was buried in the middle. Here is the general shape of what they found:

![A U-shaped curve. When the key fact sits at the start or the end of a long input, the model uses it correctly more often. When it sits in the middle, accuracy dips. A handwritten note at the bottom of the dip says easy to miss.](https://www.kirupa.com/ai_fundamentals/images/context_window_limits_lost_middle.png)

Newer models handle long inputs better than the ones in that paper, but the general lesson has held up. In July 2025, Chroma's [Context Rot](https://www.trychroma.com/research/context-rot) report tested 18 models and found that their performance became less reliable as inputs got longer, even on deliberately simple tasks.

This means a big window tells us the most the model can hold. It doesn't tell us how well the model will use all of it. If Priya's walnut allergy is sitting in the middle of a huge pile of old messages and menus, the model might technically see it and still not act on it.

The practical habit is to keep the desk tidy. Put the instructions that matter most at the start, keep the current question at the end, and leave out anything the model doesn't need for this particular answer. A smaller, cleaner context costs less, comes back faster, and is very often more accurate too.

## Conclusion

A model's context window is its whole world for one request. It holds the system prompt, the conversation, any documents or tool results, and the reply that's still to be written. Because the model doesn't remember anything between requests, our code resends the conversation every turn, which is why a chat with Biscuit keeps growing until it hits the limit.

When the window runs out, we decide what leaves the desk by trimming, summarizing, or retrieving, instead of letting something important fall off at random. And even when everything fits, a tidy context tends to get better answers, because models don't read every part of a long input equally well.

What we haven't talked about is **who** gets to put what on the desk, and which instructions win when two of them disagree. That's the job of the system prompt, and it's where we'll go next in [System Prompts vs. User Prompts](https://www.kirupa.com/ai_fundamentals/system_prompts_vs_user_prompts.htm).