# Keeping the Index Fresh

by [kirupa](https://www.kirupa.com/me/index.htm) | filed under [AI Fundamentals](https://www.kirupa.com/ai_fundamentals/index.htm)

On Tuesday morning, Trailhead finds a problem with its Ember camp stove. A batch of them has a faulty valve, so the company issues a recall. Three things change in the help center that day:

- A new page goes up: **Ember stove safety recall**, telling owners to stop using the stove and return it.
- The **Ember product page** gets a red banner about the recall.
- The old **Ember quick-start guide**, which has a cheerful section on ***your first boil***, gets deleted.

On Wednesday, Mei, who is packing for a trip, asks the help assistant:

*Is the Ember stove safe to use?*

The assistant says yes. It quotes the quick-start guide and mentions how quickly the Ember boils water.

Here is Mei's chat:

![Trailhead's help assistant chat window. Mei asks, Is the Ember stove safe to use? The assistant replies, Yes! The quick-start guide says it's great, and the Ember boils water in no time. A handwritten note flags that reply: It was recalled!](https://www.kirupa.com/ai_fundamentals/images/keeping_the_index_fresh_chat_ember.png)

Here's what went wrong. The assistant doesn't search the help center. It searches the **index**, the collection of chunks and [embeddings](https://www.kirupa.com/ai_fundamentals/embeddings.htm) we built from the help center. That index was last updated on Sunday. As far as it knows, there is no recall, the product page has no banner, and the quick-start guide is alive and well.

Here is the gap, laid out on a timeline:

![A timeline from Sunday to Wednesday. On Tuesday the help center gets a new recall notice, an edited product page, and a deleted quick-start guide. Below, the index is still Sunday's copy, with the old product page, the quick-start guide, and no recall, marked with a handwritten Stale! On Wednesday, Mei asks Is the Ember stove safe to use?, and her question goes to that stale copy.](https://www.kirupa.com/ai_fundamentals/images/keeping_the_index_fresh_stale.png)

An index is a copy, and copies drift. Keeping it fresh is mostly about noticing changes quickly and applying them correctly.

Onwards!

## What Can Change

It helps to list every way the source and the copy can disagree, because each one needs handling:

1. New documents. The recall notice exists in the help center but not in the index.
2. Edited documents. The product page changed, but the index still has the old text and the old embeddings.
3. Deleted documents. The quick-start guide is gone from the help center but still sitting in the index.
4. Permission changes. Someone is added to or removed from a group that can read a document. (This is the nasty one in [Permission-Aware Retrieval](https://www.kirupa.com/ai_fundamentals/permission_aware_retrieval.htm).)
5. Our own changes. We switch embedding models or change how we [chunk](https://www.kirupa.com/ai_fundamentals/chunk_size_and_overlap.htm). Now every chunk in the index was built the old way.

## The Simple Way: Rebuild Everything

The easiest approach is to throw the index away on a schedule and rebuild it from scratch. Read every document, chunk it, embed it, and swap the new index in. Trailhead's Sunday job does exactly this.

Rebuilding has real advantages. It handles new, edited, and deleted documents all at once with no special cases, because anything that's gone from the source simply doesn't get rebuilt.

The downside is that the index is only as fresh as the schedule. Weekly rebuilds mean information can be up to a week old, and Mei's question landed right in that gap. Rebuilding more often costs more, because we're re-embedding thousands of chunks that didn't change just to pick up the few that did.

## The Better Way: Only Process What Changed

Most systems move to **incremental updates**. Instead of rebuilding everything, we figure out what changed and only reprocess that.

There are two common ways to find changes:

- We can ask the source. Many systems can list documents changed since a given time (***give me everything updated after Sunday 2 a.m***.). We poll that list every few minutes.
- We can get told. Some systems send a notification (often called a webhook or change event) whenever a document is created, updated, or deleted. We react to each one.

When a document changes, we re-chunk it. Then, before paying to embed anything, we check whether each chunk's text actually changed. A cheap way to do that is a **content hash**: a short fingerprint computed from the chunk's text. Same text, same hash. If the Ember product page has 8 chunks and only the first one gained a recall banner, only that one chunk gets a new hash, and only that one chunk needs a new embedding.

Here's that check:

![The eight chunks of the Ember product page, with Sunday's fingerprints on the left and today's on the right. Chunks 2 through 8 have identical fingerprints and are skipped. Chunk 1 changed from a91f to e4d0 and is the only one re-embedded.](https://www.kirupa.com/ai_fundamentals/images/keeping_the_index_fresh_hashes.png)

With incremental updates, the recall notice and the edited product page can be searchable within minutes of being published, instead of waiting for Sunday.

## Deletes Are the Sneaky Part

Incremental updates are great at noticing things that exist. They're much worse at noticing things that *stopped* existing. ***Give me everything updated since Sunday*** won't mention the quick-start guide, because a deleted document doesn't show up as updated. It just isn't there anymore.

So the deleted guide keeps living in the index, still answering questions about ***your first boil***. This is about the worst kind of staleness there is. An outdated price is annoying, while a deleted safety document that keeps answering questions is dangerous. The same goes for documents deleted for legal or privacy reasons.

There are a few ways to catch deletes:

- Listen for delete events, if the source sends them.
- Have the source keep a **tombstone** (a small ***this was deleted*** record) instead of vanishing silently, so the update feed can include it.
- Periodically compare the full list of document IDs in the source against the index, and remove anything the source no longer has.

Here's a deleted document hanging around:

![Two panels. In the help center, the recall notice and the product page remain, and the quick-start guide has left an empty dashed spot marked deleted Tuesday. In the index, the quick-start guide is still there with a ghost icon and a handwritten note: Still answering!](https://www.kirupa.com/ai_fundamentals/images/keeping_the_index_fresh_ghost.png)

Edits have a smaller version of this problem. If the product page used to produce 8 chunks and now produces 6, we have to remove chunks 7 and 8. The safe habit is to treat each document as a unit: when it changes, replace **all** of its chunks, rather than adding new ones next to the old ones.

## Changing the Embedding Model

The biggest refresh of all happens when we switch embedding models. Embeddings from different models can't be compared, so we can't mix old and new in one index. Every chunk has to be re-embedded.

The usual way to do this without downtime is to build the new index alongside the old one. The old index keeps answering questions while the new one fills up. Once the new one is complete and has passed our [retrieval tests](https://www.kirupa.com/ai_fundamentals/recall_vs_precision.htm), we switch traffic over, keep the old one around briefly in case we need to switch back, and then delete it. While both exist, every change has to be applied to both, or the new index will be stale on day one.

Here is that switch-over in three stages:

![Three stages of switching embedding models. Filling: the assistant uses the old index while the new one fills up. Switched: the assistant uses the new index, and the old one is faded but kept for rollback. Cleaned up: only the new index remains.](https://www.kirupa.com/ai_fundamentals/images/keeping_the_index_fresh_switch.png)

## How Fresh Is Fresh Enough?

Not every document needs the same speed. For Trailhead, a reasonable set of targets might look like this:

| Source | Freshness target |
|---|---|
| Safety recalls and notices | minutes |
| Prices and stock | minutes to an hour |
| Help center articles | within a day |
| Employee handbook | within a week |

These are illustrative targets, not rules. What matters is that you pick them on purpose, per source, based on what goes wrong when that source is out of date.

To know whether you're hitting those targets, store two timestamps on every chunk: when the source document was last changed, and when we indexed it. The difference between them is your freshness lag, and it's worth tracking and alerting on just like errors or latency. Showing the source's last-updated date next to [citations](https://www.kirupa.com/ai_fundamentals/citation_grounding.htm) helps too, because it lets people spot an old document for themselves.

## Conclusion

Mei got a confident, wrong answer because the assistant searched a Sunday copy of a help center that changed on Tuesday. The index is a copy, and every new, edited, and deleted document, every permission change, and every embedding model upgrade is a chance for that copy to drift.

Full rebuilds are simple but only as fresh as their schedule. Incremental updates, with content hashes to skip unchanged chunks, get changes in within minutes. Deletes need their own explicit handling, edits should replace a document's chunks as a unit, and model changes need a side-by-side rebuild. Then we pick a freshness target for each source and measure the lag so we know when we've missed it.

With incremental updates in place, the recall notice was searchable a few minutes after it was published. The next time someone asks about the Ember, the assistant tells them to stop using it, and Mei is bringing a different stove.