# A/B Testing

by [kirupa](https://www.kirupa.com/me/index.htm) | filed under [AI Fundamentals](https://www.kirupa.com/ai_fundamentals/index.htm)

Let's say we have two versions of our AI assistant's prompt. Both pass our offline tests. Both look good when we try them by hand. One is a bit warmer and chattier, and the other gets straight to the point. Which one should we ship?

We could argue about it. We could poll our teammates. Or we could let real users settle it, by randomly giving half of them one version and half the other, and measuring what happens. That's an **A/B test**, and it's the closest thing we have to a fair fight between two versions of a product.

The idea is simple. Running one well takes a little more care, especially with LLMs, where two versions can also differ in cost and speed, on top of quality. Let's walk through it.

Onwards!

## The Starting Point

We'll use Sprocket, the support assistant for our online bike-parts shop. Our current prompt (we'll call it A) has been live for months. Our engineer Dana has written a new prompt (we'll call it B) that gives shorter answers and asks a clarifying question earlier when a request is vague.

B passed our [golden dataset](https://www.kirupa.com/ai_fundamentals/golden_datasets.htm) and our [regression tests](https://www.kirupa.com/ai_fundamentals/prompt_regression_tests.htm), so we know it doesn't break anything we've already thought of. What we don't know is whether customers will actually be better off with it. That's a question only real traffic can answer (the [Offline vs. Online Evals](https://www.kirupa.com/ai_fundamentals/offline_vs_online_evals.htm) article goes into why).

## Step 1: Decide What "Better" Means Before Starting

This is the step people skip, and it's the most important one. Before a single customer sees version B, we write down how we'll judge the result.

For Sprocket, ***better*** means the customer got their problem solved without needing a person. So our main metric (the primary metric) is the resolution rate. That's the share of conversations that end without the customer asking for a human agent, and without them coming back about the same issue within two days.

We also pick a few guardrail metrics. These are numbers we don't expect to improve, but we'd refuse to ship if they got worse:

- Cost per conversation, since longer conversations use more tokens.
- Response time, since customers won't wait forever.
- Thumbs-down rate, as a rough check that nobody is getting upset.

Why write all of this down first? Because if we look at twenty metrics after the test, one of them will probably look better for B just by chance, and we'll be tempted to declare victory on that one. Picking ahead of time keeps us honest. Here's Sprocket's test plan:

![A test plan for comparing prompts A and B names one primary metric to improve, resolution rate, and three guardrails that must not get worse: cost per conversation, response time, and thumbs-down rate. A handwritten note says: write this first!](https://www.kirupa.com/ai_fundamentals/images/ab_testing_plan.png)

## Step 2: Split the Traffic Randomly

Now we split customers into two groups at random. Half see A and half see B. Random assignment is what makes the comparison fair. Because a coin flip decides who gets what, both groups end up with roughly the same mix of easy and hard questions, happy and grumpy customers, and weekday and weekend shoppers. The only systematic difference left between them is the prompt.

There's one detail that matters a lot for chat assistants. We split by customer (or at least by conversation), not by message:

![Customers are split by a coin flip per customer, so each group consistently gets version A or version B. Avoid a coin flip per message, which mixes a chatty reply from A and a terse reply from B in one conversation.](https://www.kirupa.com/ai_fundamentals/images/ab_testing_split.png)

If we flipped a coin for every message, a single customer might get the chatty Sprocket for one reply and the terse one for the next. That's a confusing experience, and it muddles the results too, because we can no longer say which version a resolved conversation belongs to.

## Step 3: Wait for Enough Data

Here's where patience comes in. Let's say after the first morning, 40 conversations have gone through each version, and the numbers look like this (illustrative):

- A resolved 28 of 40 conversations (70%)
- B resolved 31 of 40 conversations (77.5%)

B is ahead! Should we ship it?

Not yet. With only 40 conversations each, a difference of 3 resolved conversations could easily be luck. If we flipped a fair coin 40 times and got 23 heads instead of 20, nobody would be surprised. Small samples bounce around a lot.

If we track the difference between B and A over the course of the test, it tends to look something like this:

![A line chart of B minus A over 14 days of a test. Early on, the measured difference swings far above and below zero, and the shaded range the true difference is likely in is very wide. By the second week the line settles slightly above zero and the range narrows. A handwritten note points at the day 1 spike: don't stop here!](https://www.kirupa.com/ai_fundamentals/images/ab_testing_peeking.png)

The trap here is called **peeking**. If we check the results every hour and stop the moment B looks ahead, we'll often stop on one of those early random swings and ship something that isn't actually better. The standard fix is to decide the sample size before starting, using a sample size calculator (most A/B testing tools have one built in), and only make the call once we've reached it. Some experimentation tools use statistical methods designed for checking results continuously, which is fine. What isn't fine is stopping whenever the numbers happen to look good.

## Step 4: Check That the Split Worked

Before reading the result, we check one boring thing. Did we actually get the split we asked for? If we asked for 50/50 and ended up with 5,000 conversations in A but only 4,400 in B, something is broken. Maybe version B errors out on some inputs and those conversations never get logged. Maybe B is slower to start and some customers leave before their conversation is recorded.

Here is the split we got:

![We asked for a 50/50 split. Version A got 5,000 conversations and version B got only 4,400. That gap is a sample ratio mismatch, so something is broken and the result can't be trusted yet.](https://www.kirupa.com/ai_fundamentals/images/ab_testing_split_check.png)

This is called a **sample ratio mismatch**, and when it shows up, the results can't be trusted, because the two groups are no longer the same mix of customers. The book *Trustworthy Online Controlled Experiments*, by Ronny Kohavi, Diane Tang, and Ya Xu, treats this as one of the first trust checks to run before believing any result. If the split is off, we find out why before we look at anything else.

## Step 5: Read the Result

Two weeks later, we've reached our planned sample size, and the split looks right. Here are the (illustrative) results:

![Results for versions A and B: resolution rate rises from 71 to 75 percent, which is better; cost per conversation rises to 1.3 times A, which is worse; response time and thumbs-down rate show no change.](https://www.kirupa.com/ai_fundamentals/images/ab_testing_results.png)

Version B resolves more conversations, and the difference held up with enough data. But one of our guardrails moved. B costs about 30% more per conversation, because its clarifying questions make conversations longer.

This is where the A/B test hands the decision back to us. The test tells us what B gains and what it costs. Deciding whether the trade is worth it is our call. Is a 4-point increase in resolution worth 30% more in model costs? If every unresolved conversation means a support agent spends ten minutes on it, the answer is probably yes. That's a business question, and now we get to answer it with real numbers instead of opinions.

## When Not to A/B Test

A/B tests aren't always the right tool. If we don't have much traffic, a test might take months to reach a reliable answer, and offline evals plus careful [error analysis](https://www.kirupa.com/ai_fundamentals/error_analysis_on_traces.htm) will teach us more, faster. If a change fixes a clear bug or a safety problem, we don't hold the fix back from half our customers just to measure it. We ship the fix.

And some effects only show up over a long time. Whether customers come to trust Sprocket more over several months won't show up in a two-week test. The reverse can happen too. A new version can do well at first simply because it's new and people are curious, which is called a novelty effect, and that boost can fade once the newness wears off.

## Conclusion

An A/B test is a fair fight between two versions. We write down what ***better*** means and what must not get worse. We split customers randomly so the two groups are alike. We wait until we have enough data instead of stopping at the first exciting swing. And we check that the split itself worked before trusting the result.

With LLM features, the guardrails matter as much as the main metric, because a version that helps customers more can also cost more or respond more slowly. A good A/B test won't make that trade-off go away, but it will put real numbers on both sides so we can make the call with our eyes open. For how the signals we collect from customers keep improving Sprocket over time, see [User Feedback Loops](https://www.kirupa.com/ai_fundamentals/user_feedback_loops.htm).