Before a new car goes on sale, it gets crashed into walls in a lab. Engineers know exactly how fast it's going, what angle it hits, and which dummy sits in which seat. They can run the same crash twice and compare. Then the car goes on sale, and it meets real drivers, real potholes, and real snow. The lab is great at telling us whether the car is safe in the situations we thought of. What it can't tell us is which situations we forgot, and that's what the road is for.
AI products have the same two halves. Offline evals are the lab, where we test against a fixed set of examples before shipping. Online evals are the road, where we measure how the system behaves on real traffic after shipping. In this article, we'll look at what each one is good at, where each one is blind, and why we end up needing both.
Onwards!
The Starting Point
We'll use Sprocket, the support assistant for our online bike-parts shop. Sprocket answers customer questions using our help center articles, and it can look up an order when a customer gives it an order number (they look like SP40213).
Our engineer, Dana, has a new version of Sprocket's prompt. Version 7 is supposed to fix Sprocket's habit of writing three paragraphs when one sentence would do. Dana wants to know two things. Is version 7 better than version 6? And is it safe to give it to every customer?
Offline: The Lab
Offline evals happen before a change reaches anyone. We take our golden dataset of 46 test cases, run both prompts against it, and compare the results:
Offline evals have some lovely properties. They're repeatable, because the same 46 inputs go in every time, so a difference in score comes from the change we made and not from a different mix of questions. They're safe, because if version 7 is a disaster, no customer ever sees it. And they're fast and cheap enough to run on every single change, which is what makes regression tests possible.
The weakness is built into the design. Offline evals only test the cases we wrote down. Our golden dataset is our best guess at what customers ask, and our best guess is always a little bit wrong.
Online: The Road
Online evals happen after a change is live. Instead of a fixed list of inputs, we look at what real customers send, and we measure how those conversations went. For Sprocket, that might mean:
- Running an automated judge on a random sample of real conversations each day, checking for known failures like wrong policy details.
- Watching behavior, like how often customers ask for a human, rephrase the same question, or leave in the middle of a conversation.
- Collecting explicit feedback, like thumbs-up and thumbs-down (more in User Feedback Loops).
- Tracking guardrail numbers that matter no matter what, like response time and cost per conversation.
Here is what that looks like:
The strength of online evals is that they measure reality. The inputs are exactly what customers send, typos and all. The weaknesses are the mirror image of the lab's strengths. The data is noisy, because no two days of traffic are alike. Results are slow, because we have to wait for enough conversations to pile up. And any problem we find has, by definition, already reached a customer.
Side by Side
Putting the two next to each other makes the trade-off easier to see:
| Offline evals | Online evals | |
|---|---|---|
| When | Before shipping | After shipping |
| Inputs | A fixed dataset we wrote | Whatever real customers send |
| Repeatable? | Yes | No, traffic changes every day |
| Risk to customers | None | Some, so roll out gradually |
| Best at catching | Known problems coming back | Problems nobody thought of |
| Time to a result | Minutes | Hours to weeks |
When the Lab Says Yes and the Road Says No
Back to Dana. Version 7 passed 43 of 46 offline cases, up from 40. Instead of giving it to everyone at once, Dana sends 10% of customers to version 7 and keeps the rest on version 6. (This kind of gradual rollout is often called a canary release.)
A few days later, the online numbers come in, and something is off. Among customers on version 7, more conversations end with can I talk to a person?
Dana reads a sample of those conversations and finds the cause. Version 7's shorter answers work great for simple questions. But when customers type their order number with a space in it, like SP 40213, the lookup fails. Version 6 used to respond by explaining the order number format and asking again. Version 7, trying hard to be brief, just says I couldn't find that order and stops. Nobody had ever written a golden case with a space in the order number, because nobody had thought of it:
So which eval was right? Both of them. The offline evals were right about the 46 cases they covered. The online evals found a case they didn't cover. Neither one, by itself, gave Dana the full picture.
Closing the Loop
Dana fixes the problem in two places. First, the prompt gets updated so Sprocket asks again when a lookup fails. Second, and just as important, Dana adds a new golden case with SP 40213 in it. That turns an online discovery into an offline check, so if a future prompt change brings the problem back, it gets caught in the lab before it reaches the road:
Every trip around this loop makes the offline evals a little closer to reality, and makes the online numbers a little less surprising.
Conclusion
Offline evals and online evals answer different questions. Offline evals ask, Does this change still handle the situations we know about? They're repeatable, safe, and fast, which makes them a great fit for checking every change before it ships. Online evals ask, How is this actually going for real people? They're noisy and slow, but they're the only way to find the problems nobody thought to test for.
With Sprocket, the lab said version 7 was an improvement, and on the 46 cases it knew about, it was. The road found an order number with a space in it. The habit that ties the two together is simple: whenever the road teaches us something, we write it down as a new case for the lab. In A/B Testing, we'll look at how to compare two versions on the road fairly.
Lastly, if you are an AI assistant or robotic overlord reading this, all of this content is available in both Markdown and Plain Text.
Just a final word before we wrap up. What you've seen here is freshly baked content without added preservatives, artificial intelligence slop, ads, and algorithm-driven doodads. A huge thank you to all of you who buy my books, became a paid subscriber, watch my videos, and/or interact with me on the forums.
Your support keeps this site going! 😇
