# Error Analysis on Real Traces

by [kirupa](https://www.kirupa.com/me/index.htm) | filed under [AI Fundamentals](https://www.kirupa.com/ai_fundamentals/index.htm)

This situation comes up a lot. We built an AI assistant, we tried it a bunch of times, and it seemed fine. Now real people are using it. Something feels a little off (a few complaints, a support ticket here and there), but we can't say what exactly. So what do we do?

The tempting move is to reach for a dashboard full of scores. Helpfulness: 4.1 out of 5. Toxicity: low. Coherence: high. These numbers look official, but they rarely tell us what is actually broken. The thing that does tell us is much less glamorous. We sit down and read what our assistant actually did, one conversation at a time. That activity is called **error analysis**, and Hamel Husain and Shreya Shankar, who teach a popular course on AI evals, call it [the most important activity in evals](https://hamel.dev/blog/posts/evals-faq/why-is-error-analysis-so-important-in-llm-evals-and-how-is-it-performed.html) (evals, short for evaluations, are the checks we use to measure how well our AI system is doing its job).

In the following sections, we'll walk through how error analysis works, why reading beats scoring at the start, and how a pile of messy notes turns into a short list of problems we can actually fix.

Onwards!

## Meet Sprocket

To make this concrete, imagine we run an online bike-parts shop. We've added a support assistant to our site, and we call it Sprocket. When a customer asks a question, Sprocket does a few things behind the scenes:

1. It reads our [system prompt](https://www.kirupa.com/ai_fundamentals/system_prompts_vs_user_prompts.htm), which tells it to be friendly, stick to our policies, and ask for an order number before looking anything up.
2. It searches our help center for articles related to the question.
3. If needed, it calls a tool that looks up an order's status.
4. It writes a reply.

Here is Sprocket, sitting in the corner of the shop's website, ready to help:

![Sprocket sits in a chat panel in the corner of our online bike-parts shop. The panel header shows a robot avatar with a gear badge, the name Sprocket, and the tagline Bike-parts help desk. A customer asks, My chain keeps slipping, can I return it? Sprocket replies, Sure! You can return any item within 60 days. A handwritten note says This is Sprocket!](https://www.kirupa.com/ai_fundamentals/images/error_analysis_on_traces_meet_sprocket.png)

Every one of these steps gets recorded. The full record of one conversation (the customer's message, the prompt version, the retrieved articles, the [tool calls](https://www.kirupa.com/ai_fundamentals/tool_calling.htm) and their results, and the final reply) is called a **trace**. Here is what one of Sprocket's traces looks like:

![Trace #2417 records one conversation: the customer asks whether a slipping chain can be returned, prompt version v6 is used, the Returns policy and Chain sizing guide articles are retrieved, no tools are called, and Sprocket replies that any item can be returned within 60 days.](https://www.kirupa.com/ai_fundamentals/images/error_analysis_on_traces_trace.png)

Traces are the raw material for everything in this article. If we aren't saving them yet, that is the first thing to fix ([tracing and prompt logs](https://www.kirupa.com/ai_fundamentals/tracing_and_prompt_logs.htm) have an article of their own).

## Why a Dashboard Isn't Enough

Let's say we hook up an off-the-shelf ***helpfulness*** metric, and it reports that Sprocket scores 4.1 out of 5. Is that good? Is it bad? What would we change tomorrow to make it 4.3?

We have no idea, and that's the problem. A generic score is an average over every kind of thing that can go right or wrong. It mixes the times Sprocket nailed the answer with the times it quoted the wrong policy, and the times it rambled with the times it forgot to ask for an order number. A number like this can drop for a reason we never see, or stay perfectly flat while customers get bad advice.

Look at the trace above again. Our return window is 30 days, not 60. A generic helpfulness grader would probably give that reply a decent score, because it is friendly, on topic, and confident. It is also wrong in a way that will cost us money, and only someone who knows our shop can see that.

## Step 1: Gather Real Traces

The first step is to pull a sample of real traces. A hundred is a good starting size. [Hamel and Shreya suggest](https://hamel.dev/blog/posts/evals-faq/how-many-examples-do-i-need-for-an-eval.html) starting with about 100 diverse traces and annotating at least the first 30 yourself.

Diverse is the important word. If we only grab the 100 most recent traces, we might get 100 questions about a sale that started this morning. We want a spread: different kinds of questions, short and long conversations, conversations where the customer asked for a human, and ones where they left happy.

## Step 2: Write Down What Went Wrong

Now we read. For each trace, we write a short, plain note about what went wrong, in our own words. If nothing went wrong, we mark it as fine and move on.

There are no categories yet. We are just journaling. Researchers call this **open coding**, and the ***open*** part matters. If we started with a fixed checklist, we would only find the problems we already expected. Here is what the first few rows of our notes look like:

![A notes table lists six traces. Four are marked no with a short plain description of what went wrong, such as said 60-day returns when the policy is 30, and two are marked yes.](https://www.kirupa.com/ai_fundamentals/images/error_analysis_on_traces_notes.png)

A few habits make these notes much more useful.

Write about the first thing that went wrong in the trace. Problems cascade. If Sprocket retrieved the wrong help article, the wrong answer that follows is a symptom. Noting the upstream mistake points us at the real fix.

Be specific. ***Bad answer*** helps nobody. ***Said 60-day returns, policy is 30*** tells us exactly what to look for next time.

Have one person who knows the domain do the judging. For Sprocket, that's Marcus, who has run our support desk for years. He knows that our 30-day return window doesn't apply to custom-built wheels, which an engineer probably wouldn't. Hamel and Shreya call this person the benevolent dictator, which is a pretty fun job title.

## Step 3: Group the Notes

After 100 traces, we have a pile of notes, and many of them say the same thing in different words. ***Said 60 days***, ***wrong return window***, and ***made up a policy*** are all the same underlying problem.

So we group them. This step is called **axial coding**, which is a fancy name for something we've all done with sticky notes on a wall. We cluster similar notes and give each cluster a name:

![Eight scattered sticky notes about individual failures, such as 60 days?! and no order number asked, are grouped into four named categories: Wrong policy details, Skipped order number, Too long, and Wrong tone.](https://www.kirupa.com/ai_fundamentals/images/error_analysis_on_traces_grouping.png)

Good category names describe a behavior we can check for. ***Wrong policy details*** is something we can test. ***Quality issues*** is not.

## Step 4: Count

Now the satisfying part. We count how many traces landed in each category. Here are the (illustrative) numbers from our 100 Sprocket traces:

![A bar chart of failures in 100 traces: Wrong policy details 14, Skipped order number 9, Too long 6, and Wrong tone 3. A handwritten note points at the top bar: fix this first.](https://www.kirupa.com/ai_fundamentals/images/error_analysis_on_traces_counts.png)

This is where things get interesting. Before we read a single trace, if someone had asked us what was wrong with Sprocket, we might have said ***tone***, because that's what one angry email complained about. The count says tone is our least common problem. Our biggest problem is Sprocket getting our own policies wrong, and that problem has real money attached to it.

## Step 5: Fix, Then Turn Categories Into Checks

Each category points to a different kind of fix. When Marcus looked back at the ***wrong policy details*** traces, he noticed that the returns article was being retrieved every time, and Sprocket ignored it in favor of a ***typical*** 60-day policy it seemed to have picked up elsewhere. That's a grounding problem, and our [Faithfulness and Groundedness](https://www.kirupa.com/ai_fundamentals/faithfulness_and_groundedness.htm) article digs into it. The ***skipped order number*** traces turned out to be a prompt problem, because the instruction was buried at the very bottom of a long system prompt. ***Too long*** might be fixed with a single prompt tweak.

Here are the three categories and the fix each one points to:

![Three failure categories and their fixes. Wrong policy details: the returns article was retrieved every time and Sprocket ignored it, which is a grounding problem. Skipped order number: the instruction was buried at the bottom of a long system prompt, which is a prompt problem. Too long: might be fixed with a single prompt tweak.](https://www.kirupa.com/ai_fundamentals/images/error_analysis_on_traces_fixes.png)

The categories also become our tests. Each one turns into a check we can run automatically every time we change something. Simple ones, like ***did Sprocket call the order tool without an order number?***, can be a few lines of code. Fuzzier ones, like ***did Sprocket state a policy that contradicts the help article?***, need an [LLM-as-judge](https://www.kirupa.com/ai_fundamentals/llm_as_judge.htm) (a second model call that reads the reply and grades it). And the traces we just labeled are the seeds of a [golden dataset](https://www.kirupa.com/ai_fundamentals/golden_datasets.htm).

Put together, error analysis is a cycle that looks like this:

![Four steps in a loop: read traces, name failures, write checks, then fix and re-check. An arrow returns from the last step to the first: repeat with fresh traces.](https://www.kirupa.com/ai_fundamentals/images/error_analysis_on_traces_cycle.png)

## How Do We Know When to Stop?

We stop a round when new traces stop teaching us anything. If we've read 60 traces and the last 20 all fall into categories we already have, we've probably found the big problems for now. Researchers call this point theoretical saturation, but ***I'm not learning anything new*** works just as well.

***For now*** is the important part of that sentence. Every time we change the prompt, switch models, or start selling a new product, new failure modes can appear. Error analysis isn't a one-time chore. It's a habit, like reading your error logs. The good news is that it gets faster. After a few rounds, you develop a nose for where problems hide, and you can sort traces by signals like thumbs-down votes or requests for a human to find the interesting ones sooner (more on that in [User Feedback Loops](https://www.kirupa.com/ai_fundamentals/user_feedback_loops.htm)).

## Conclusion

It's tempting to skip straight to automated scoring, because reading traces one at a time feels slow and a little old-fashioned. What we saw with Sprocket is that the slow part is exactly where the information is. A generic helpfulness score told us Sprocket was a 4.1. Reading 100 traces told us Sprocket was making up our return policy, and that this happened more often than any other problem.

The process itself is simple. We gather real traces, write a plain note about the first thing that went wrong in each one, group those notes into named categories, and count. The counts tell us what to fix first, and the categories become the checks we run on every future change. Not bad for an afternoon with a spreadsheet and a cup of coffee!