# Tracing and Prompt Logs

by [kirupa](https://www.kirupa.com/me/index.htm) | filed under [AI Fundamentals](https://www.kirupa.com/ai_fundamentals/index.htm)

When a regular web feature breaks, it usually tells us. A request throws an exception, a status code turns red, and a stack trace points at line 212. LLM features have a sneakier way of failing. The request succeeds, the status code is a cheerful 200 OK, and the answer is wrong. Nothing in our usual logs looks unusual, because from the server's point of view, nothing went wrong.

To debug that kind of failure, we need to see what the model actually saw, what it said back, and everything that happened in between. That record has two parts: traces and prompt logs. Let's look at why we need both by chasing down one wrong answer.

Onwards!

## The Mystery of the Shipped Order

Imagine we run the support assistant for Spokes, a small online bike shop. On Tuesday, a customer named Maya asks where her new helmet is. Our assistant tells her it shipped yesterday and will arrive Thursday. Thursday comes and goes. No helmet. Maya writes in, understandably annoyed, and our engineer Raj discovers that the order never shipped at all.

Raj's first move is to look at what he has. He has Maya's question, and he has the assistant's answer, and that's it:

![Maya asks where her helmet is. Her question goes into a closed box, a wrong answer comes out saying it shipped yesterday, and the only signal from the system is a successful 200 OK.](https://www.kirupa.com/ai_fundamentals/images/tracing_and_prompt_logs_black_box.png)

Which part went wrong? Did the assistant look up the wrong order? Did it read an outdated shipping policy? Did the order system return bad data? Did the model just make something up? Every one of those is plausible, and the box in the middle won't tell us which one happened.

## Why LLM Apps Are Extra Hard to Debug

With ordinary code, Raj could often reproduce the bug by running the same input again. With our assistant, that rarely works. For one thing, the prompt the model saw doesn't exist anywhere in our source code. It was assembled at the moment Maya asked, from a [system prompt](https://www.kirupa.com/ai_fundamentals/system_prompts_vs_user_prompts.htm) (which might have been edited last week), a chunk of retrieved documentation, the earlier part of the conversation, and the results of any tools the model called. Change any one of those, and it's a different prompt.

Here are the pieces that prompt was built from:

![The prompt the model saw was assembled at the moment Maya asked, from four pieces: a system prompt that might have been edited last week, a chunk of retrieved documentation, the earlier part of the conversation, and the results of any tools the model called. Change any one of those, and it's a different prompt.](https://www.kirupa.com/ai_fundamentals/images/tracing_and_prompt_logs_assembled.png)

Even if Raj could rebuild that exact prompt, the model's output isn't deterministic, so running it again can produce a different answer. ***I tried it and it worked for me*** proves very little. On top of that, the data behind the tools keeps changing. By the time Raj looks, Maya's order may have actually shipped, and asking again gets a perfectly correct answer that tells him nothing.

So we can't count on reconstructing what happened after the fact. We have to write it down as it happens.

## Traces: The Story of One Request

A trace is the record of everything that happened while handling one request. Each step in that work is a span. A span has a name, a start time, an end time, some attributes, and a parent, so the spans nest inside each other like a family tree. Every span in the trace shares the same trace ID, which is what ties them together.

Here is what the trace for Maya's message looks like when we lay its spans out on a timeline:

![A trace waterfall for Maya's request. Inside the handle_message span, the system retrieves the shipping policy, calls the model, runs the get_order tool, which returns label_created, and calls the model again to write the answer.](https://www.kirupa.com/ai_fundamentals/images/tracing_and_prompt_logs_waterfall.png)

This already narrows things down a lot. The assistant did look up the right order, A-1042. The order system answered correctly. The status was `label_created`, which in our shop means we printed a shipping label and the box is still sitting on a shelf. Somewhere between that tool result and the final answer, `label_created` turned into ***it shipped yesterday***.

To see how, Raj needs to look inside the last model call.

## Prompt Logs: What the Model Actually Saw

A prompt log is the exact text that went into a model call and the exact text that came out, along with the details of the call. A good prompt log for one model call records:

- the full input, meaning every message in order, including the system prompt and any tool results
- the full output, including any [tool calls](https://www.kirupa.com/ai_fundamentals/tool_calling.htm) the model asked for
- the model name and version the provider reports back
- the settings, like [temperature](https://www.kirupa.com/ai_fundamentals/temperature_predictable_output.htm) and the maximum output length
- which version of our prompt template was used
- input and output token counts, the finish reason, and the latency

When Raj opens the prompt log for that last model call, the problem jumps right out:

![The prompt log shows the model received the system prompt, a policy line saying orders usually ship within one business day, Maya's question, and a tool result with status label_created dated the day before. It turned that into the claim that the order shipped yesterday.](https://www.kirupa.com/ai_fundamentals/images/tracing_and_prompt_logs_prompt_log.png)

There's the culprit. The model saw a status it had never been told about (`label_created`), a date that was yesterday, and a policy snippet saying orders usually ship within a business day. It connected those dots in the most natural way, and the result was wrong. Nobody wrote a bug, exactly. The system prompt just never explained what our order statuses mean.

Now Raj has a real fix to choose from. He can teach the system prompt the difference between `label_created` and `in_transit`, or he can change the tool to return a plain-language status like ***packed, not yet picked up by the carrier***. Better still, Maya's conversation can become a [regression test](https://www.kirupa.com/ai_fundamentals/prompt_regression_tests.htm), so a future prompt change that reintroduces this mistake gets caught before it ships.

## Making This Work in Practice

Raj's investigation worked because the trace and the prompt log were there when he needed them. Making that true every time comes down to a few practical decisions.

The first decision is what shape our traces take, and a standard shape beats one we invent. OpenTelemetry, the widely used open standard for traces and metrics, has [semantic conventions for generative AI calls](https://github.com/open-telemetry/semantic-conventions-genai). They name a model call span something like `chat support-model` and record attributes like `gen_ai.request.model`, `gen_ai.usage.input_tokens`, and `gen_ai.usage.output_tokens`. At the time of writing, those conventions are still marked as in development, but following them means our traces work with lots of existing tools instead of only our own.

It also helps to make traces easy to find. If we show a short reference code in the support chat, or store the trace ID with the conversation, then when Maya writes in, Raj can jump straight to her trace instead of searching by timestamp.

The trickiest decision is how to handle the data itself, because prompt logs are sensitive. They contain everything users type, which includes names, addresses, order numbers, and occasionally things people really should not have typed into a support chat. Here is the path a trace should take before it lands in storage:

![Before traces are stored, personal data is redacted. Then they are sampled, keeping every error but only 10 percent of everything else. The trace store itself is locked down and deletes traces after 30 days.](https://www.kirupa.com/ai_fundamentals/images/tracing_and_prompt_logs_pipeline.png)

That's where [PII Redaction](https://www.kirupa.com/ai_fundamentals/pii_redaction.htm) comes in, stripping personally identifiable information (PII), like names and addresses, out of our traces before they're stored. The OpenTelemetry conventions also treat capturing the actual prompt and response text as optional, partly because of these privacy concerns and partly because modern prompts can be enormous. A long conversation with a few retrieved documents can easily be tens of thousands of tokens per call, so many teams store the full text separately and keep only a reference to it in the trace.

We also don't need to keep every trace. Nobody will ever look at the thousandth successful ***What are your store hours?*** conversation. We do want every error, every timeout, every thumbs-down, and every conversation that ended with ***talk to a human***.

## Conclusion

When we started, Raj had a question, an answer, and a closed box in the middle. A trace opened that box and showed him the sequence of steps: the retrieval, the model calls, and the tool call that returned `label_created`. The prompt log for the last model call showed him exactly what the model was looking at when it decided Maya's helmet had shipped. Together, they turned ***the AI is wrong sometimes*** into a specific, fixable cause.

What traces don't give us is the big picture. They tell the story of one request at a time, which is perfect for debugging Maya's helmet and not so great for answering ***why did our bill double this month?*** For that kind of question, we roll the numbers in our traces up into [Token and Cost Metrics](https://www.kirupa.com/ai_fundamentals/token_and_cost_metrics.htm).