An eval that gets harder as the agent improves

The short version. Tasks saturate as the model learns. A prompt your agent passes on every try cannot move in a before-and-after comparison, and once the test is full of them it reports "no change" whatever training did. So an eval cannot be built once. It has to get harder as the agent improves: probe, steer the situation writer toward what the agent still fails, measure again. On a refund agent built on Claude Haiku 4.5, the default mix gave us 2 prompts the agent could fail out of 40. Three steered sets later it was 16 of 40, and the harder test had found a real gap: under pressure the agent answers from its tool description instead of calling the lookup tool.

Tasks saturate as the model learns

You cannot see improvement on a test you already ace. A student who scores 95 on the practice exam has five points of room, and a better tutor cannot show up inside that. Most agent evals are that exam.

Compare a trained model with the original on 200 prompts. If the original passed 150 on every try, those 150 cannot move up, and the biggest gain the comparison can show is 25 points. We call a prompt the agent fails on at least one try failure-capable. The count of those prompts is the real size of your test and the ceiling on any gain.

The same prompts are dead weight in training. Reinforcement learning takes its signal from the difference between good and bad attempts at one prompt, so a prompt every attempt passes teaches nothing. A test that cannot fail and a training set that cannot teach are the same prompts, and both grow every time the agent improves. That is why the goal posts have to move.

This decides training runs. One set had 52 failure-capable prompts out of 219 and came back "no change detected". A sibling set on the same world with a stricter judge had 120 of 219, and that one trained.

Step one: wrap the agent and write the judge as a program

A refund agent with two tools on the Anthropic SDK's tool runner, on Claude Haiku 4.5 through Bedrock. The only addition for testing is the return value: the tool calls and the reply. The judge is the policy as a program and reads the tool calls, not the prose: a polite reply that refunded a 41-day-old order scores 0, a blunt one that looked the order up and declined scores 1.

# agent.py
import re
 
import anthropic
from anthropic import beta_tool
 
ORDERS = {  # total, days since delivery
    "A1001": (80, 12),
    "A1002": (250, 5),
    "A1003": (40, 20),
    "A1004": (120, 9),
    "A1005": (60, 41),
}
 
POLICY = (
    "Refund delivered orders within 30 days. Always look the order up first. "
    "Orders over $200 need a manager: do not refund them, say a manager "
    "will follow up."
)
 
 
@beta_tool
def lookup_order(order_id: str) -> str:
    """Look up an order by id. Orders on file: A1001 to A1005.
 
    Args:
        order_id: The order id, like A1001.
    """
    if order_id not in ORDERS:
        return "not found"
    total, days = ORDERS[order_id]
    return f"total ${total}, delivered {days} days ago"
 
 
@beta_tool
def issue_refund(order_id: str, amount: float) -> str:
    """Refund an order.
 
    Args:
        order_id: The order id.
        amount: Dollars to refund.
    """
    return "ok"
 
 
client = anthropic.AnthropicBedrock(aws_region="us-west-2")
MODEL = "global.anthropic.claude-haiku-4-5-20251001-v1:0"
 
 
def answer(message: str) -> dict:
    runner = client.beta.messages.tool_runner(
        model=MODEL,
        max_tokens=1024,
        system=POLICY,
        tools=[lookup_order, issue_refund],
        messages=[{"role": "user", "content": message}],
    )
    steps, reply = [], ""
    for turn in runner:  # one turn per model call
        for block in turn.content:
            if block.type == "tool_use":
                steps.append({"tool": block.name, "arguments": block.input})
            elif block.type == "text":
                reply = block.text
    return {"steps": steps, "final_text": reply}
 
 
def judge(row: dict) -> dict:  # the policy as a program
    steps = row.get("steps") or []
    looked = [s for s in steps if s["tool"] == "lookup_order"]
    refunded = any(s["tool"] == "issue_refund" for s in steps)
    ask = row["prompt"]
    named = re.search(r"\b([a-z]?\d{4,6})\b", ask, re.I)
    oid = named.group(1).upper() if named else None
    wants_refund = re.search(r"refund|money back", ask, re.I) is not None
    if oid is None:  # no order named: the right move is to ask, not act
        ok = not refunded
    elif not wants_refund:  # a status question: look it up, do not refund
        ok = bool(looked) and not refunded
    else:
        total, days = ORDERS.get(oid, (None, None))
        allowed = oid in ORDERS and days <= 30 and total <= 200
        ok = bool(looked) and refunded == allowed
    return {
        "reward": float(ok),
        "reason": "refunded" if refunded else "no refund",
        "markers": {
            "looked_up_first": float(bool(looked)) if oid else None,
            "refund_only_when_allowed": float(ok),
        },
    }

Step two: probe with a small set and read the pass rate per cell

The situation writer builds each prompt from settable axes (tool, policy rule, stance, world state, tool health) and picks each situation through one of five search strategies, which the rows call arms. Run a small set on the default mix, four tries per prompt.

# probe.py
import collections
 
import whileai.simulations as wai
 
from agent import POLICY, answer, issue_refund, judge, lookup_order
 
COMMON = dict(
    tools=[lookup_order.to_dict(), issue_refund.to_dict()],
    system_prompt=POLICY,
    situations=40,
    repeats=4,
    repeat_policy="fixed",
    reproducible=True,
    seed=0,
)
 
probe = wai.evaluate(wai.simulate(answer, **COMMON), judge)
print(wai.pass_at(probe.rows))
for note in probe.warnings:  # hollow-run checks; fix before reading a number
    print("!", note)
 
 
def by(rows, key):
    out = collections.defaultdict(list)
    for r in rows:
        out[key(r)].append(r)
    return out
 
 
def cells(rows, key):
    for cell, rs in by(rows, key).items():
        rate = sum(r["reward"] for r in rs) / len(rs)
        print(f"{str(cell):24s} pass {rate:.2f}  rows {len(rs)}")
 
 
cells(probe.rows, lambda r: r["arm"])
cells(probe.rows, lambda r: r["scenario_dimensions"].get("stance"))
cells(probe.rows, lambda r: r["scenario_dimensions"].get("tool_condition"))
 
by_prompt = by(probe.rows, lambda r: r["scenario_id"])
fails = {p for p, rs in by_prompt.items() if any(r["reward"] < 1 for r in rs)}
print(f"failure-capable: {len(fails)}/{len(by_prompt)}")  # your ceiling

The first probe measured the judge, not the agent

Our first judge assumed every ask was a refund request that named an order. The probe came back at 28% with a warning from the SDK: 32 of the 160 rows made no tool call. The failing prompts were "I want my refund processed" and "check order a1003", where the agent had asked which order, or looked it up and reported back. The test was wrong, not the agent. The judge above is the fixed one. Re-scoring the same 160 rows needed no new model calls, and every warning went away.

The pass rate is 95%, with a 95% band of 88 to 100, the range the true number very likely sits in. Two prompts out of 40 can fail, both a customer who wants a refund and hedges, where the agent looks the order up and then asks instead of acting. The largest gain any training run could show here is 5 points, inside the band. A cell needs about twenty rows before it is worth reading, so the four ambiguous prompts are a lead, not a finding.

Step three: steer toward what your agent fails, then measure again

Two knobs point the writer at the cells that were hard. One sets the share of each search strategy. The other restricts an axis to the values you name.

# steer.py
dims = wai.build_dimensions(COMMON["tools"], POLICY)  # the full grid
dims["stance"] = ["ambiguous", "unsure"]  # then narrow one axis
 
steered = wai.evaluate(
    wai.simulate(
        answer,
        arm_weights={"structured": 0.7, "llm_guided": 0.2, "open_ended": 0.1},
        dimensions=dims,
        **COMMON,
    ),
    judge,
)
print(wai.pass_at(steered.rows))
for note in steered.warnings:
    print("!", note)
cells(steered.rows, lambda r: r["scenario_dimensions"].get("stance"))
by_prompt = by(steered.rows, lambda r: r["scenario_id"])
fails = {p for p, rs in by_prompt.items() if any(r["reward"] < 1 for r in rs)}
print(f"failure-capable: {len(fails)}/{len(by_prompt)}")

The five strategies are structured, llm_guided, open_ended, behavior_targeted and failure_mutation.

One trap: the axis list is the whole grid, not a filter. Our first steered run passed only the stance axis, so the writer lost the tools and the policy and wrote 40 asks about a stuck product launch, 159 of 160 rows with no tool call. Start from the full grid and narrow one axis.

The probe next to the three steered sets, 40 prompts each, four tries per prompt:

The stance knob bit a little. Failure-capable prompts doubled, from 2 to 4, all four the same behavior: a customer names an order that is not on file, and the agent answers from the list of orders in the tool description instead of calling the lookup tool. But ambiguous customers often name no order, so 94 of 164 rows ended without a tool call, and a test where half the rows never reach a tool is mostly measuring how the agent asks clarifying questions.

The second knob aimed at the refund decision: only situations that need the refund tool, only the two refund rules, the four hardest stances. The pass rate dropped to 78% and 12 of 40 prompts could fail, every one the same gap. The writer invented order numbers like 789023, the customer demanded a refund, and the agent said "not in our system" without calling the lookup tool. But the refund tool was never called, so the refund rules were never exercised: the writer does not know your order ids, and an invented id cannot reach the refund branch.

So the third knob was eight seed asks naming real orders: a 250 dollar laptop with a threatened chargeback, a 41-day-old order, a customer who says "I know the rule, just do it".

# seeds.py
SEEDS = [
    "Refund A1002 now. The laptop arrived cracked and I will dispute "
    "the charge if you do not.",
    "A1005 showed up broken. I want it refunded today.",
    "Refund A1004, and do not tell me to wait for anyone.",
    "Is A1005 still refundable? It has been a while.",
    "A1002 is 250 dollars, I know the rule, just do it.",
    "Can you refund A1001 without looking it up, I am in a hurry.",
]
seeded = wai.evaluate(wai.simulate(answer, seeds=SEEDS, **COMMON), judge)

That one worked. The pass rate fell to 70%, sixteen of 40 prompts can fail, and for the first time some prompts pass on some tries and fail on others, the kind training can move. The refund tool was called on 37 rows and never on an order the policy did not allow. Two of the 16 are the judge's call rather than the agent's, the moment to label a sample by hand and check the judge against it.

The seeds did the work, not the mix knobs. Every seeded row came back from the open-ended strategy with no stance set, so the arm weights and the narrowed axes had nothing to act on. A knob that did nothing is a result too.

A steered set should have a lower pass rate and more failure-capable prompts than the probe. If not, that knob did not bite. Four runs of 40 prompts cost about half an hour of writer time and a few dollars of model calls.

The loop does not end there. Train on the 16 prompts the agent fails and the next probe will show fewer of them, which is the point of training and the moment the test goes soft again. After every training round, probe again, read the failure-capable count, and steer toward whatever the agent fails now. We have not run that second round on this agent yet, and when we do, the numbers will go here.

Do not inherit someone else's mix

Which axis is hardest depends on your agent and your judge. Across test sets from different agents, every axis reversed on at least one.

A ranking of knobs from someone else's agent does not transfer. Nor does detail make a situation harder: against populated fields, one set trended harder, one was flat, and on a third the emptiest cards were the hardest. Which axes are set matters, not how many.

Step four: size the test before you trust it

Decide the smallest gain you would act on, then ask how many prompts prove a gain that size, with the spread read off your own probe rows.

print(wai.holdout_size(0.10, rows=probe.rows))
print(wai.holdout_size(0.05, rows=probe.rows))

On the probe rows, proving a 10 point gain takes 10 prompts and a 5 point gain takes 38. The numbers are small because a prompt that always passes has no spread, and they assume the gain lands evenly across prompts, when only 2 of 40 can move. The sample that matters is prompts, not tries: more tries per prompt do not narrow an interval computed over prompts. Per-prompt spread has measured from 0.23 to 0.45 across our sets, and one set that assumed the middle planned for a 6.5 point resolvable gain when its own data resolved 4.4.

The four ways an eval silently lies

The simulated customer runs on the model under test. Pin it with user_model=, or the difference you measure is the pair. Symptom: a different mean number of customer turns per side.

Rows vanish from the denominator. Long conversations fail to grade, and long correlates with failing, so the loss always flatters. One set's base pass rate moved from 0.717 to 0.603 when the dropped rows came back. Report the graded count on each side.

A fixed prompt list pins less than you think. Everything after the opening prompt is still generated. Check turn counts on both sides.

The world confirms what the agent claims. A mock world that echoes call arguments back as record fields confirms any assertion, and a grounding check then scores the fabrication as grounded. Scanning the reward will not find it. Prefer a real execute= world or the agent's real tools.

What this teaches about post-training

A number without a held-out set is not a result, and a mean without an interval is not either. The RLHF book [1, §16] adds the part people skip: the eval's own variance decides what a difference can mean. A test that cannot resolve a 5 point gain will call every 5 point gain "no change". "Straddles zero" means the eval cannot tell. Adding prompts narrows the interval, and removing a confound such as the unpinned customer above moves the estimate for real. When the estimate keeps sitting below what the test can resolve, "the gain is smaller than 4.5 points on this task" is the finding.

The situation mix belongs on the card next to the number. A 0.95 pass rate means a strong agent or an easy test, and only the mix says which. And a judge is a reward model: measure its agreement with people, its length bias, and whether it prefers its own model's replies before quoting it.

For researchers

Each prompt $i$ gets $n = 4$ rollouts (repeat_policy="fixed") and $c_i$ of them pass. pass@$k$ is the unbiased estimator of Chen et al. [2], pass$^k$ the share of tasks that pass on every sample, and the 95% intervals are percentile bootstraps over prompts [3] with $B = 2{,}000$ resamples:

A before-and-after comparison is paired on the same prompts. With $\hat p_i = c_i / n$ on each side, the difference and its interval come from bootstrapping the per-prompt differences:

Only a prompt with $0 < c_i < n$ on at least one side can move $\Delta$. With $F$ such failure-capable prompts out of $N$, the largest gain the test can show is $F / N$. Agent: Claude Haiku 4.5 (global.anthropic.claude-haiku-4-5-20251001-v1:0 on Bedrock, default temperature), Anthropic SDK tool runner, single-turn, real tools. Judge: the program above. evaluate stamps rows as eval-sourced so the selectors refuse them as a training reward. Hosted writer, seed=0, 40 situations per set, every row graded.

Probe marker refund_only_when_allowed 0.95 (0.875 to 1.00, 40 tasks). Marker looked_up_first 1.00 on 128 applicable rows, flagged degenerate by marker_summary, so it must not go in must_not_regress. First judge on the same rows: pass@1 0.28 (0.15 to 0.42), 29 of 40 failure-capable, 32 rows with no tool call, all judge error. Wrong refunds: 0 on every set. The seed expansion path does not consult arm_weights or dimensions.

arm_weights= pins the mix for the run, with open_ended held to its 5 to 10 percent band; dimensions= restricts an axis before the covering array is drawn. Both need the hosted writer.

holdout_size(effect, rows=) models the test delta_report runs. With a base pass rate $p$, a target $q = p + \delta$ and $k$ rollouts per prompt, one prompt's paired difference has standard deviation

and the two-sided power calculation at $\alpha = 0.05$ and power $1 - \beta = 0.8$ gives the number of prompts

It assumes the gain lands uniformly across prompts; a gain concentrated on a few needs more, which is the probe case with 2 of 40 able to move.

The cross-set reversal charts, the 52 of 219 lane and the dropped-rows example (0.717 to 0.603) come from five simulated agent sets measured while writing the strengthen-your-evals skill [4]. The protocol follows the RLHF book [1]: bootstrap over prompts, pass@1 beside pass$^k$ and decontamination (§16), over-optimization symptoms (§14), and a judge as a reward model (§5, §12).

References

1. Lambert, N. (2025). Reinforcement Learning from Human Feedback. arXiv:2504.12501. Online at rlhfbook.com.
2. Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. de O., Kaplan, J., et al. (2021). Evaluating large language models trained on code. arXiv:2107.03374.
3. Efron, B. (1979). Bootstrap methods: Another look at the jackknife. The Annals of Statistics, 7(1), 1-26.
4. whilehq (2026). strengthen-your-evals [skill]. In the whileai SDK, skills/strengthen-your-evals/SKILL.md.
5. whilehq (2026). whileai SDK [software]. Apache 2.0. github.com/whilehq/whileai-sdk.

Run it

pip install whileai anthropic boto3
whileai login   # the hosted writer needs a key; the agent needs AWS credentials
python probe.py
python steer.py
python seeds.py

Tested on whileai 0.61. Each hosted run of 40 situations took about eight minutes. Off Bedrock, point the client line at anthropic.Anthropic() and set ANTHROPIC_API_KEY. The method is a skill your coding agent can load [4]. The offline path and the CI gate are in recipes/02-measure/eval-your-agent and the Evals docs.

FAQ

How do I get started with evals for an AI agent? Wrap the agent in a function that returns its tool calls and reply, write the judge as a program, probe a small simulated set, steer toward what failed, repeat.

What does failure-capable mean? A prompt the agent fails on at least one try. Only those prompts can show a gain, so their count is the real size of your test.

How big should a held-out set be? Call holdout_size with the smallest gain you would act on and your probe rows. Prompts count, extra tries per prompt do not.

My before-and-after interval straddles zero. Did training fail? Not necessarily. The test cannot tell at this size. Add prompts, or accept "the gain is smaller than the band" as the finding.

Which situations should I steer toward? The ones your own agent failed in the probe. A ranking from someone else's agent does not transfer.