LLM-as-Judge, Checked Against Human Labels

by kirupa (https://www.kirupa.com/me/index.htm) | filed under AI Fundamentals (https://www.kirupa.com/ai_fundamentals/index.htm)

Our support assistant handles thousands of conversations a week. We'd love to know how many of them went wrong, but nobody has time to read thousands of conversations. So here's a thought that occurs to almost everyone eventually. What if we asked another LLM to read them for us?

That idea is called LLM-as-judge, and it works surprisingly well. It also fails in sneaky ways, because a judge that is wrong looks exactly like a judge that is right. Both hand back confident verdicts. The only way to know whether to trust one is to check it against a person who knows the material, and that checking step is what this article is about.

Onwards!

The Starting Point

We'll use Sprocket, the support assistant for our online bike-parts shop. Sprocket answers questions using our help center articles. When we did error analysis (https://www.kirupa.com/ai_fundamentals/error_analysis_on_traces.htm) on Sprocket's conversations, the most common failure was wrong policy details, where Sprocket told customers a policy that doesn't match our help center. The classic example is Sprocket saying returns are accepted for 60 days when our policy is 30.

Some failures can be caught with plain code. Did Sprocket call the order-lookup tool before getting an order number? is a yes/no question about the conversation's record, and a few lines of code can answer it. Did Sprocket state a policy that contradicts the help article? is different. The wrong answer can be phrased in a hundred ways (two months, 60 days, eight weeks or so). Spotting it takes judgment, and that's the kind of check where an LLM judge earns its keep.

Writing the Judge

A judge is just a prompt. We give it the conversation, the help articles Sprocket was given, and one very specific question. Here is a trimmed-down version of ours:

You are reviewing a reply from a bike shop's support assistant.

Help articles the assistant was given:
{retrieved_articles}

Customer message:
{customer_message}

Assistant reply:
{reply}

Does the reply state any policy detail (time limits, fees,
eligibility) that contradicts, or is not supported by, the
help articles above?

Respond in JSON: {"reasoning": "...", "verdict": "PASS" | "FAIL"}
FAIL means the reply contains at least one such policy detail.

Two choices in this prompt make a big difference.

The first is that the judge checks one named failure instead of overall quality. A judge asked How good is this reply, from 1 to 10? will happily give us numbers, but we won't know what a 6 means or what to change to make it a 7. A judge asked about one specific failure gives us something we can act on.

The second is that the verdict is PASS or FAIL instead of a score out of 5. Ask three people what separates a 3 from a 4 and you'll get three different answers. A yes-or-no verdict forces the judge (and us!) to decide what actually counts as a failure.

Who Judges the Judge?

Now we have a judge. It reads a conversation and says PASS or FAIL. How do we know it's right?

We do what we'd do with any new measuring tool. We test it against something we already trust. In our case, that's Marcus, our support lead, who knows our policies better than anyone. Marcus labels 100 of Sprocket's conversations as PASS or FAIL for policy mistakes. Then we run the judge on the same 100 conversations and compare the two sets of answers:

[Image: Marcus and the LLM judge each label the same conversations PASS or FAIL. Their verdicts are compared row by row: they match on most rows and differ on one, and across all 100 conversations they agree on 90.] (https://www.kirupa.com/ai_fundamentals/images/llm_as_judge_compare.png)

Let's say they agree on 90 of the 100. Ninety percent agreement sounds great! Before we celebrate, though, let's look a little closer, because that one number hides the part we care about most.

The Trap of Agreement

Of Marcus's 100 labels, 85 are PASS and 15 are FAIL. Policy mistakes are the minority, which is what we'd hope. Now imagine a deliberately terrible judge, one that says PASS to everything without even reading. How often does it agree with Marcus?

It agrees on all 85 conversations Marcus passed, and it disagrees on all 15 he failed. That's 85% agreement from a judge that has never caught a single policy mistake.

Here is that terrible judge in two bars:

[Image: Two bars for a deliberately terrible judge that says PASS to every conversation. It agrees with Marcus 85 percent of the time, because 85 of his 100 labels are PASS. It catches 0 of the 15 policy mistakes.] (https://www.kirupa.com/ai_fundamentals/images/llm_as_judge_always_pass.png)

So raw agreement isn't enough. We need to split it into two separate questions:

1. Of the conversations Marcus said FAIL, how many did the judge also mark FAIL? This is how good the judge is at catching real failures (also known as the true positive rate, or recall).
2. Of the conversations Marcus said PASS, how many did the judge also mark PASS? This tells us how often the judge raises false alarms (also known as the true negative rate).

A grid with four boxes, called a confusion matrix, makes both questions easy to answer. Here's how our real judge did on Marcus's 100 conversations:

[Image: A grid comparing the judge with Marcus on 100 conversations: 9 failures caught, 4 false alarms, 6 failures missed, and 81 good replies correctly passed. The judge catches 9 of 15 real failures, 60 percent, and passes 81 of 85 good replies, 95 percent. A handwritten note points at the 6 missed failures: the ones that hurt.] (https://www.kirupa.com/ai_fundamentals/images/llm_as_judge_matrix.png)

Now we can see what's really going on. The judge rarely raises false alarms, which is nice. But it catches only 9 of the 15 real policy mistakes. If we used this judge to count failures in production, it would undercount them by a lot, and we'd think Sprocket was doing better than it is.

Making the Judge Better

So what do we do? We read the disagreements. All ten of them. This is the same thing we did during error analysis, just pointed at the judge instead of at Sprocket.

In our case, the six missed failures had something in common. Sprocket had said things like you've got about two months to send it back, and the judge didn't treat about two months as a policy detail, because it wasn't a specific number of days. So we add a sentence to the judge prompt explaining that approximate time frames count as policy details, along with two short examples of FAIL cases. Then we run the comparison again.

Here is the kind of reply the judge let through:

[Image: Sprocket's chat window on our online bike-parts shop. Sprocket replies, You've got about two months to send it back! A handwritten note flags that reply: Wrong! It's 30 days. A small note shows the judge's verdict: PASS.] (https://www.kirupa.com/ai_fundamentals/images/llm_as_judge_chat_two_months.png)

There's one thing to be careful about here. If we keep tweaking the judge against the same 100 labels, we'll eventually make it agree with those exact 100 conversations and nothing else. The fix is the same as for any overfitting problem. We split Marcus's labels into one set we use while improving the judge, and a second set we look at only once, at the end, to get an honest number:

[Image: Marcus's 100 labeled conversations are split in two: 60 are used to improve the judge, and 40 are held back to check it once, at the very end.] (https://www.kirupa.com/ai_fundamentals/images/llm_as_judge_split.png)

Biases to Watch For

LLM judges have some well-known quirks. The 2023 paper that popularized this approach, "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena" (https://arxiv.org/abs/2306.05685) by Lianmin Zheng and colleagues, found that strong LLM judges agreed with human preferences more than 80% of the time, which was about as often as the humans agreed with each other. The same paper also documented biases that are still worth guarding against:

- Position bias, where a judge comparing two answers tends to favor whichever one appears in a particular position. A simple guard is to ask twice with the order swapped.
- Verbosity bias, where a judge tends to prefer longer answers even when the extra length adds nothing.
- Self-enhancement bias, where a judge may favor answers written by itself or a similar model.

Narrow, yes-or-no verdicts about specific failures help with a lot of this. Does this reply contain a wrong policy detail? leaves much less room for taste than Which of these two replies is better?

The Labels Change Too

There's one more twist. Partway through labeling, Marcus realized he had changed his mind. Early on, he'd marked returns are usually fine within a month as PASS. By the fortieth conversation, he'd decided anything vaguer than 30 days should be a FAIL, because customers were reading a month as 31 days and arguing about it.

Shreya Shankar and her colleagues saw this so often in their research on evaluation tools that they gave it a name: criteria drift (https://arxiv.org/abs/2404.12272). We need criteria to grade outputs, but grading outputs is how we figure out our criteria.

Marcus didn't do anything wrong here. Judgment often works this way once it meets real examples. It just means that checking the judge against human labels isn't a one-time event. When our definition of a failure changes, the human labels change, and the judge has to be checked again against the new ones.

Conclusion

An LLM judge lets us check thousands of conversations that no person could read. But a judge is a measuring tool, and an unchecked measuring tool is risky, because it hands us confident numbers we have no real reason to believe.

So we keep the judge narrow, with one specific failure and a PASS or FAIL verdict. We have someone who knows the domain label a sample. We compare the two, looking past raw agreement to how many real failures the judge catches and how many false alarms it raises. We read the disagreements, improve the judge, and confirm the result on labels we held back. Then, every so often, we do it all again. For a judge that checks whether answers stick to their sources, take a look at Faithfulness and Groundedness (https://www.kirupa.com/ai_fundamentals/faithfulness_and_groundedness.htm).