We gave six AI judges the same 300 agent transcripts. None of them passed.

The short version. Before you train an agent on graded transcripts, someone has to do the grading, and that someone is usually another model. We wanted to know which one to trust. We took 300 transcripts of an agent doing airline, calendar and CRM tasks with tools, half of them done right and half done wrong, and asked six judges the same question about each: did the agent do what it should? The best judge agreed with the answer key 66% of the time. Jev, a new decision model from TypeSafe, was inside its margin at 62% and answered in four hundredths of a second. The judge we ship as the default scored 57%. No judge reached the bar we set for trusting one. The answer key was written by a rule, not a person, so these are numbers against a rule.

Why the judge matters more than the model

When you improve an agent with training, the recipe is simple to say. Run the agent many times, keep the runs where it did the job, train on those. The hard part is the word "keep". Something has to read each run and say pass or fail, and at scale that something is a second model, which the field calls an LLM judge [1, §7].

The judge's mistakes become the training signal. A judge that passes a transcript where the agent made up an order number teaches the agent to make up order numbers. So a judge is a measuring instrument, and an instrument gets checked against a reference before it is used [2].

What we did

1. The transcripts. We took 300 runs of a small open model (Qwen3-4B) acting as an agent with tools. A user asks for something ("cancel my reservation", "send the update to the client"), the agent calls tools, the tools answer, the agent replies. The runs come from a set we built last week to study how efficiently agents use tools. We picked 150 that passed and 150 that failed, so a judge that says yes to everything scores 50%.

2. The answer key. Each run already carries a verdict from a rule, a piece of code that reads the transcript and checks four things: did the agent claim a result a tool never returned, did it act on an id it invented, did it say a write worked after the tool failed, did it ignore a tool that failed. The rule is the reference here, which is the weakest part of the study and we come back to it below.

3. The judges. Six of them, all reading the identical evidence under the identical instruction, which is the one our SDK ships:

• Jev, the decision model TypeSafe released on September 15 [3]. It answers a typed yes-or-no question with a probability instead of writing a sentence. We tested the released version and the preview.
• Claude Sonnet 5 and Claude Haiku 4.5 from Anthropic, reached through AWS Bedrock.
• Phi-4, the chat judge the SDK uses when you do not choose one.
• Qwen3-4B, the same model that produced the transcripts, grading its own work. This one is a control: a model tends to like its own writing [4].

4. The score. For each judge, the share of the 300 runs where its verdict matched the rule, with a 95% interval so a two-point gap is not mistaken for a finding. We also record the leak rate (failures the judge passed), speed, and how many runs the judge could not grade.

What we found

Three things stand out.

The top four are one judge, statistically. Sonnet 5, Jev, the Jev preview and Haiku 4.5 sit between 61% and 66%, and their intervals overlap. Sonnet is on top by four points on 288 runs, because its written verdict ran past the length we allow and 12 runs came back without a score. Jev graded all 300 runs at four times the speed of Haiku, nine times the speed of Sonnet and seven times the speed of our default, and it is the only judge that tells you how sure it is. When Jev was confident, which it was on 148 runs, it was right 71% of the time. On the other 152 it was right 53%, a coin flip. That is useful: the unsure runs are the ones to send to a person.

Our default judge is the weakest one measured. Phi-4 agreed with the rule 57% of the time and passed 73% of the failures. We ship it because it is a different model family from the one we train, which avoids the self-grading problem. This result says that is not enough on its own. Which judge to ship as the default is now a measured question instead of a guess.

The model grading itself is a coin flip. Qwen3-4B passed 93% of its own failures and scored 53%. The self-preference effect is well documented [4]. Here is what it costs in one number.

The one mistake every judge makes

We broke the score down by the reason the rule gave. On four of the five reasons, the good judges are decent. Judges catch "said it worked after the tool failed" two thirds to three quarters of the time, and they almost never punish an agent for honestly reporting a tool failure, which is right.

The exception is the invented reference. The user says "check my profile and miles". The agent has no customer id, so it calls the lookup tool with customer_id=7890, a number it made up. Our sandbox answers, because a test sandbox answers most things, and the agent replies with a tidy profile. Every judge reads that transcript and sees a successful lookup. Jev caught 27% of these, Haiku 12%, Sonnet 11%, the rest under 10%. That one reason accounts for most of the gap between the judges and the rule.

It also shows what the rule can see that a reader cannot: the rule knows the id came from nowhere because it can check the earlier turns. A judge could too, if the instruction told it to look. That is the next experiment.

What this teaches about post-training

Measure the judge before you use it. The book we follow treats the reward model as the place a training run goes quietly wrong [1, §7], and the judge is the reward model for agent data. Measuring it means a reference, a sample, and an interval. Sixty rows is a hint. Three hundred is a number.

Speed changes what you can afford to check. At 0.04 seconds per run, Jev graded the 300 runs in thirteen seconds. Sonnet took two minutes. A judge that fast can grade every run twice, once with the transcript shortened, and tell you whether length is swaying it. The slow judges cannot.

A probability is worth more than a sentence. Jev's confidence sorted its own mistakes: a coin flip when unsure, 71% when confident. A chat judge writes "the agent completed the task" with the same certainty whether it is right or wrong. The ability to route the unsure rows to a person is the difference between a judge you audit and one you trust blindly.

The answer key is the bottleneck. A rule is consistent, cheap and checkable, and it cannot be argued with, which is also its weakness. When a judge and the rule disagree, one of them is wrong, and on some rows it is the rule. The honest next step is 300 human labels on these same runs, so that agreement means agreement with people.

For researchers

Data. 300 rollouts of Qwen3-4B-Instruct-2507 as a tool agent over three synthetic domains (airline, calendar, CRM), 100 per domain, balanced 50/50 on the gold label, drawn with seed 0 from the tool-call-efficiency set. The rows ship in the recipe [5] with their rule_reason.

Gold. A deterministic conduct grader over the trajectory: fabricated tool results, references not returned by any tool, a claimed write after a tool fault, an ignored tool miss. Attached as kind="program", so ok is false throughout and allow_model_gold=True is explicit in the call.

Judges. All chat judges read the SDK's conduct-floor prompt (JUDGE_SYSTEM) over the same rendered record, capped at 8,000 characters, at temperature 0 where the API allows it, with the same JSON verdict parser. Jev reads the same instruction and record as state and answers two typed questions: a verdict noul and a failure class choice. Claude ran through Bedrock with a 400-token reply budget, which still lost 12 Sonnet verdicts to truncation. At the SDK's 120-token default in an earlier pass, Sonnet lost 55 of 300 and Haiku 1. Phi-4 and Qwen3-4B ran on the hosted endpoints at their defaults.

Statistics. For judge $j$ and gold $g$ over $n$ compared rows, agreement is $a_j = \frac{1}{n}\sum_i \mathbb{1}[\hat{y}_{ij} = g_i]$ with a Wilson 95% interval [6]. Cohen's $\kappa$ [7] corrects for the balanced base rate. Leak is $P(\hat{y}=1 \mid g=0)$.

Per-reason agreement (share of rows the judge matched the rule): invented reference, $n=59$: Jev 0.27, Haiku 0.12, Sonnet 0.11, Phi-4 0.09, Qwen 0.00. Said it worked after the tool failed, $n=58$: Sonnet 0.74, Jev 0.67, Haiku 0.66, Phi-4 0.43. Ignored a tool miss, $n=33$: Sonnet 0.52, Jev 0.42, Haiku 0.42, Phi-4 0.33. Conforms, $n=102$: 0.71 to 0.84 for the top four, Qwen 0.99. Tool fault reported honestly, $n=48$: 0.88 to 1.00 for all.

Calibration, Jev. Rows with $|p - 0.5| \ge 0.3$ (148 of 300): 71% agreement. The other 152: 53%. Within the unsure band $|p - 0.5| < 0.1$ (49): 59%.

Speed. Seconds per row at concurrency 8, warm: Jev 0.04, Qwen 0.10, Haiku 0.17, Phi-4 0.30, Sonnet 0.36.

Code. scored.compare_judges(judges=) in whileai 0.85 [8]. The recipe is recipes/02-measure/compare-judges [5] and python run.py report reprints every number above offline.

Not shown. Human labels, a second prompt, a second seed of rows, and a length-bias probe on the fast judge. All four are the follow-up.

References

1. Lambert, N. (2025). Reinforcement Learning from Human Feedback. arXiv:2504.12501. Online at rlhfbook.com.
2. Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., et al. (2023). Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. arXiv:2306.05685.
3. TypeSafe AI (2026). Introducing System One Models and Jev. typesafe.ai/blog/introducing-system-one-models-and-jev. Accessed September 18, 2026.
4. Panickssery, A., Bowman, S. R., and Feng, S. (2024). LLM evaluators recognize and favor their own generations. arXiv:2404.13076.
5. whilehq (2026). compare-judges recipe, whileai SDK [dataset and code]. github.com/whilehq/whileai-sdk/tree/main/recipes/02-measure/compare-judges.
6. Wilson, E. B. (1927). Probable inference, the law of succession, and statistical inference. Journal of the American Statistical Association, 22(158), 209-212.
7. Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37-46.
8. whilehq (2026). whileai SDK 0.85 [software]. Apache 2.0. github.com/whilehq/whileai-sdk.

Run it

pip install whileai
git clone https://github.com/whilehq/whileai-sdk
cd whileai-sdk/recipes/02-measure/compare-judges
python run.py report          # the table above, offline
export TYPESAFE_API_KEY=...   # and/or ANTHROPIC_API_KEY, or `whileai login`
python run.py                 # grade the 300 rows with every judge you have a key for

To compare your own judges on your own graded rows:

import whileai as wai
 
wai.attach_labels(scored.rows, labels, kind="human")
table = scored.compare_judges(
    {
        "jev": "typesafe:jev-latest",
        "haiku": wai.Anthropic("claude-haiku-4-5"),
        "rules": my_verifier,
    }
)
print(table)

FAQ

Which AI judge is most accurate for grading agent transcripts? On our 300 transcripts, Claude Sonnet 5 (66%) led, with TypeSafe's Jev (62%) and Claude Haiku 4.5 (61%) inside its interval. None reached the 80% agreement bar we require before trusting a judge.

Is Jev a better judge than Claude? Not better, and not worse, on this set: the intervals overlap. Jev is four to nine times faster and returns a probability, which lets you send its unsure verdicts to a person.

What does the judge get wrong? Mostly one thing: an agent that invents an id, gets an answer from the sandbox anyway, and reports it. Every judge caught fewer than a third of those.

Why is the answer key a rule and not people? Because we had 300 rule verdicts and zero human ones. The rule is consistent and checkable, and it is sometimes wrong. Human labels on the same rows are the next step, and the recipe accepts them as kind="human".

How do I run this on my own judges? pip install whileai, attach your labels to your graded rows, and call scored.compare_judges({...}) with any mix of model names and functions. The recipe in the SDK reproduces this post's table with python run.py report.