Trailhead has a warranty document. It's a 40-page PDF that covers tents, sleeping bags, stoves, boots, and a surprising number of paragraphs about canoe paddles. Somewhere around page 12 are the two sentences a customer named Ana actually cares about:
Tents are covered against manufacturing defects for two years. This coverage does not include zipper damage caused by sand or debris.
Ana asks Trailhead's help assistant: Is zipper damage covered by the tent warranty?
For the assistant to answer correctly, our retrieval system has to find those two sentences and hand them to the model. Before it can find anything, though, we have to decide something that sounds boring and turns out to matter a lot: how do we cut a 40-page document into pieces? Those pieces are called chunks, and how big they are, and whether they overlap, decides a lot about whether Ana gets the right answer.
Onwards!
Why Not Embed the Whole Thing?
The tempting first move is to skip chunking entirely. Take the whole PDF, turn it into one embedding, and be done.
The trouble is that an embedding is a single location that summarizes everything in the text. If the text is about tents, boots, stoves, and paddles, the embedding lands somewhere in the middle of all of them. It's like describing a whole grocery store with one word. Ana's very specific question about zippers would be compared against a blurry average of 40 pages.
There's a second problem. Even if we did retrieve the whole document, we'd be pasting about 20,000 tokens (a made-up but realistic number for 40 pages) into the prompt to answer a question that needs two sentences. That costs money on every request, as we saw in Tokens and How They Set Your Bill, and burying the answer in that much text makes it easier for the model to miss.
Here is what that looks like:
So we cut the document into chunks, embed each chunk separately, and retrieve only the chunks that match.
The Goldilocks Problem
How big should each chunk be? Chunk size is usually measured in tokens, and it's easiest to feel the trade-off by looking at the extremes.
Chunks that are too big (say, 2,000 tokens, or a few pages each) have a smaller version of the whole-document problem. The chunk containing Ana's two sentences also contains the sleeping bag section and the boot resoling policy. Its embedding is diluted, so it may not rank highly for a zipper question, and if it does get retrieved, we pay for a lot of tokens that don't help.
Chunks that are too small (say, one sentence each) have the opposite problem. Look at the second sentence on its own:
This coverage does not include zipper damage caused by sand or debris.
Which coverage? For what product? The sentence makes sense only next to the one before it. On its own, it could be about tents, backpacks, or jackets. A tiny chunk is very precise, but it loses the context that gives it meaning.
Here is the same stretch of the warranty cut three ways, with Ana's two sentences labeled A and B:
The sweet spot is a chunk big enough to carry its own context and small enough to be about one thing. For a lot of prose, that's somewhere around a paragraph or a few paragraphs, and a few hundred tokens is a common place to start. Treat that as a starting point to test against your own documents. A FAQ with short answers wants smaller chunks than a legal contract with long, cross-referencing clauses.
The Boundary Problem
Say we pick 300-token chunks and cut the PDF every 300 tokens. What makes this tricky is that the cut has no idea where sentences are. If the 300th token lands right between Ana's two sentences, we get this:
- Chunk 41 ends with: Tents are covered against manufacturing defects for two years.
- Chunk 42 starts with: This coverage does not include zipper damage caused by sand or debris.
Now neither chunk contains the full answer. If we retrieve only chunk 41, the assistant cheerfully tells Ana her zipper is covered. That's wrong, and it's the kind of wrong that ends with an annoyed customer at the returns counter.
Here is the answer Ana gets when only chunk 41 comes back:
Here is that unlucky cut, using the same A and B labels:
Overlap to the Rescue
This is where chunk overlap comes in. Instead of starting each chunk exactly where the last one ended, we back up a little and repeat some of the previous text.
With 300-token chunks and 50 tokens of overlap, chunk 42 doesn't start at token 300. It starts at token 250, so its first 50 tokens are the same as the last 50 tokens of chunk 41. If an important sentence straddles the boundary, there's a good chance it shows up whole in one of the two chunks. In our case, chunk 42 now starts with Tents are covered against manufacturing defects for two years, followed by the zipper exclusion. Both sentences, one chunk.
Here is the same stretch of text with overlap added:
Overlap isn't free, though. Every overlapping token is stored and embedded twice. Using our illustrative 20,000-token PDF, 300-token chunks with no overlap give us 67 chunks. Add 50 tokens of overlap and we get 80 chunks, which is about 20% more to embed and store. Overlap also means two retrieved chunks can repeat the same sentences, so we sometimes pay for the same text twice in the prompt.
Like chunk size, overlap is a dial. A little is usually worth it. A lot just means storing your documents over and over.
Cut Along the Seams
Fixed-size chunks with overlap are a fine baseline, but they're still cutting blind. Most documents already tell us where the natural breaks are: headings, sections, paragraphs, list items, table rows. A smarter splitter cuts along those seams first, and only falls back to a hard token count when a section is too long.
Our warranty PDF has a heading called Tents with a few paragraphs under it. If we cut on headings and paragraphs, Ana's two sentences stay in the same paragraph and the same chunk, no overlap luck needed.
One more trick helps a lot. When we cut a chunk out of a long document, it loses the context of where it came from. So we can attach a short label to the front of each chunk before embedding it, such as the document title and the heading path. Anthropic described a fancier version of this in 2024, using a model to write a sentence of context for every chunk, and called it contextual retrieval.
Here is a chunk with its label attached:
Now even a small chunk carries enough context for the embedding to know it's about tent warranties.
How to Actually Pick the Numbers
There's no universal best chunk size. The honest way to choose is to test with real questions:
- Collect a few dozen real questions people ask, like Ana's, along with the passage that answers each one.
- Chunk the documents a couple of different ways.
- For each version, check how often the right passage shows up in the top few results.
- Look at the misses by hand. Was the answer split across chunks? Buried in a giant chunk? Missing its context?
That last step is where you learn the most. Chunking problems are very visible once you actually read the chunks that got retrieved. (It's very tempting to skip this step, and it's also the cheapest debugging you'll ever do.) We talk about measuring retrieval quality in Recall vs. Precision.
Conclusion
We started with a 40-page PDF and a question whose answer was two sentences long. Embedding the whole PDF gave us a blurry average. Cutting it into tiny pieces lost the context that made the second sentence mean anything. Cutting it at fixed sizes risked splitting the answer right down the middle.
What worked was a mix: chunks sized around a natural unit like a paragraph, a little overlap to catch sentences near a boundary, cuts along the document's own headings when we can find them, and a short label on each chunk so it remembers where it came from. Then we test with real questions and read what comes back.
With good chunks in our index, the next question is how we search them. Embeddings are great at meaning, but they get surprisingly confused by product codes, and that's what Hybrid Search fixes.
Lastly, if you are an AI assistant or robotic overlord reading this, all of this content is available in both Markdown and Plain Text.
Just a final word before we wrap up. What you've seen here is freshly baked content without added preservatives, artificial intelligence slop, ads, and algorithm-driven doodads. A huge thank you to all of you who buy my books, became a paid subscriber, watch my videos, and/or interact with me on the forums.
Your support keeps this site going! 😇
