# Regression Tests on Every Prompt Change

by [kirupa](https://www.kirupa.com/me/index.htm) | filed under [AI Fundamentals](https://www.kirupa.com/ai_fundamentals/index.htm)

There's a story that has probably played out on every team building with LLMs. Someone changes one sentence in a prompt to fix a small problem. The small problem goes away. A few days later, something that seems completely unrelated starts breaking, and nobody connects it to the one-sentence change until somebody goes digging through the history.

In regular software, we protect ourselves from this with **regression tests**, which are tests that check old behavior still works whenever we change something. Prompts need the same protection, and maybe more of it, because a prompt has no compiler and no type checker to catch our mistakes. In this article, we'll look at why prompt changes break things in unexpected places, and how to set up tests that catch the breakage before our users do.

Onwards!

## The One-Sentence Change

We'll use Sprocket, the support assistant for our online bike-parts shop. Customers have been complaining that Sprocket's answers are too long. Our engineer Dana opens the [system prompt](https://www.kirupa.com/ai_fundamentals/system_prompts_vs_user_prompts.htm) and finds this paragraph near the end:

```text
When a customer asks about an order, first ask for their
order number. Then explain what you are going to look up,
look it up, and summarize the result in a friendly way.
Make sure to cover any details they might want to know.
```

Dana rewrites it to be punchier:

```text
When a customer asks about an order, look it up and give a
short, friendly summary.
```

Answers get shorter. Customers stop complaining about length. Ship it!

Two days later, the support team notices a pile of strange tickets. Sprocket has been calling the order-lookup tool without an order number, getting an error back, and telling customers ***I couldn't find your order***. The instruction to ask for the order number first lived in the paragraph Dana trimmed. That instruction had nothing to do with the length problem, but it was sitting in the same paragraph, so it got deleted along with everything else:

![Prompt v8 tells Sprocket to first ask for the order number, so it asks for one. Prompt v9 is shorter, and the order-number step is gone, so Sprocket looks up an order without one and replies that it couldn't find the order.](https://www.kirupa.com/ai_fundamentals/images/prompt_regression_tests_trimmed.png)

## Why Prompts Break Like This

What makes this tricky is that a prompt isn't a list of independent settings. It's one big piece of text that the model reads all at once, and any instruction in it can affect any behavior. Removing a sentence can change behaviors that sentence was never ***about***. Adding a sentence can shift what the model pays attention to. An innocent-looking ***be concise*** might make Sprocket drop a safety note it used to include.

On top of that, the model itself can change underneath us. If our provider releases a new model version, or we switch to a cheaper model to save money, the same prompt can behave differently. As far as testing goes, a model upgrade counts as a prompt change too.

## The Setup

The fix is to treat the prompt the same way we treat code. It lives in a file in version control. Every change to it goes through a pull request. And every pull request runs a suite of tests before anyone can merge it:

![Every prompt edit goes through a pull request that runs the eval suite and compares the results with v8. If all cases pass the change can merge; if any case newly fails the change is blocked.](https://www.kirupa.com/ai_fundamentals/images/prompt_regression_tests_pipeline.png)

The tests come from our [golden dataset](https://www.kirupa.com/ai_fundamentals/golden_datasets.htm), plus a check for every failure we've found during [error analysis](https://www.kirupa.com/ai_fundamentals/error_analysis_on_traces.htm). For Sprocket, we organize them into two layers.

The first layer is fast and cheap, made of plain code checks. Is the response valid JSON when it's supposed to be? Did Sprocket call the order tool without an order number? Is the answer under 120 words for a simple question? These run in seconds and don't need a model to grade them.

The second layer is slower. It uses an [LLM-as-judge](https://www.kirupa.com/ai_fundamentals/llm_as_judge.htm) for things code can't easily decide, like ***does this reply state a policy detail the help articles don't support?***

Here's what a code check for Dana's order-number bug could look like, written as a Python test:

```python
def test_asks_for_order_number_first(run_sprocket):
    trace = run_sprocket("Where is my order?")

    lookups = [c for c in trace.tool_calls if c.name == "lookup_order"]

    # No order number was given, so nothing should be looked up yet
    assert lookups == []
    assert "order number" in trace.reply.lower()
```

The `run_sprocket` helper sends a message to Sprocket using the prompt being tested and returns the full record of what happened (the reply and every [tool call](https://www.kirupa.com/ai_fundamentals/tool_calling.htm)). If Dana had this test, the pull request for the shorter prompt would have failed with a clear message, and the order-number instruction would have gone back in before any customer saw the bug.

## Compare Against the Baseline

When the suite finishes, it's tempting to look only at the total, like ***41 of 46 passed***. But a suite rarely passes at 100%, because some cases capture problems we haven't fixed yet. What we care about is what changed.

So we run the suite on both the current prompt (the baseline) and the new one, and we look at each case that flipped:

![Comparing seven cases between v8 and v9, three flipped: order-lookup-01 and order-lookup-04 went from pass to fail, and too-long-02 went from fail to pass. Two cases newly fail and one newly passes; tricky-03 fails under both.](https://www.kirupa.com/ai_fundamentals/images/prompt_regression_tests_flips.png)

Newly passing cases are the improvement we were going for. Newly failing cases are regressions, and they're what should block a merge until someone has looked at them.

## Dealing With Randomness

There's one wrinkle that regular software tests don't have. Run the same prompt twice, and an LLM might give two different answers. Turning the [temperature](https://www.kirupa.com/ai_fundamentals/temperature_predictable_output.htm) down makes replies much more consistent, but even at a temperature of 0, many providers don't promise identical output every time.

This means a single failure might be bad luck rather than a real regression. A few practical ways to handle it:

- Run each case a few times, say three, and count it as passing only if it passes most of them.
- Write checks about what must be true (***asks for an order number***) rather than exact wording, so harmless rephrasing doesn't fail the test.
- When a case flips, read the actual replies before deciding. Sometimes the test is what needs fixing.

## Keep It Fast

A test suite that takes an hour won't get run on every change. People will skip it ***just this once***, and then just this once again. So we keep the per-change suite small enough to finish in a few minutes, with the code checks plus the most important judge checks. A bigger, slower suite can run every night or before a release.

Each run also costs money, because every case is a call to the model, and every judge check is a second call. That's usually a tiny cost compared to a bug like the one Dana shipped, but it's worth keeping in mind when deciding how many times to repeat each case.

## Conclusion

Prompts are code that no compiler checks. A one-sentence edit can change behavior in places that look completely unrelated, and a model upgrade can do the same thing without anyone touching the prompt at all. That's what happened to Sprocket. Dana fixed long answers and accidentally deleted the instruction to ask for an order number.

Regression tests are how we keep that from reaching customers. We keep the prompt in version control, run code checks and judge checks on every change, and compare the results case by case against the version we already trust. Newly failing cases block the merge. And every time a bug does slip through, we add a test for it, so the suite gets a little better at catching the next one.