Agent evals on frontier models

The short version. Once your agent passes a test every time, that test can no longer show a gain or catch a regression. This is the ten-minute path to finding the failures nobody wrote a test for, on the model you already run, with your own key. The library writes situations your agent has never met, reports the pass rate with a 95% interval, and fails the pull request when a named behavior gets worse.

Install and probe

uv add whileai

import whileai as wai
from agent import POLICY, TOOLS, judge  # your tools, system prompt and judge
 
# Your model, your key. Or wai.OpenAI(...), "ollama:...", any callable.
wai.configure(agent=wai.Anthropic("claude-haiku-4-5"))
 
data = wai.simulate(tools=TOOLS, system_prompt=POLICY, situations=40, repeats=4)
scored = data.grade(judge)  # a verifier, a rubric judge, or any callable
print(scored.pass_at)
# pass@1 0.95 [0.88..1.00] | pass^4 0.95 | headroom 0.05 (40 groups, k=4)

Apache 2.0, on GitHub. Runs on your machine with your key. simulator=False runs offline with no key at all.

What you get

Situations, not prompts. The writer combines your tools, the state of the world and the customer's mood so that every pair of settings shows up at least once (a covering array). The world fails on purpose: a tool times out, a customer will not cooperate.

pass@1 with a 95% interval. The share of first tries that pass, over four tries per situation, with the range the true number very likely sits in. A mean without that range is not a result.

pass^4. The share of situations passed on all four tries. The rest can still fail, and only those can show a gain. At 1.0 the test has nothing left to teach.

A check that fails the build. Main and the candidate run the same situations, and you get the difference with its interval. Name the behaviors that must not get worse, and the check exits 1 when one does.

On every pull request

common = dict(tools=TOOLS, system_prompt=POLICY, repeats=4)
before = wai.simulate(main_agent, **common).grade(judge)
after = wai.simulate(candidate_agent, tasks=before.rows).grade(judge)  # same k
 
report = wai.compare(
    before.rows,
    after.rows,
    target="pass_at_1",  # paired delta, bootstrap over situations
    must_not_regress=["looked_up_first", "refund_only_when_allowed"],
)
# PASS | NO DIFFERENCE | FAIL | NOT COMPARABLE
print(report["headline_verdict"])
sys.exit(0 if report["ok"] else 1)

What it found

A refund agent on Claude Haiku 4.5 passed 95% of the default mix, and only 2 of 40 situations could fail.

Three steered sets later, 16 of 40 could, and 37 of 160 runs had never called a tool.

The full run, with the code for each set, is in Evals that get harder.

When you are ready

The scored rows are post-training data. scored.select(mode="rl") keeps the situations the agent passes some of the time, the ones RL can learn from, and rows.export() feeds TRL or a GRPO environment on Modal or Prime Intellect with your own keys. How the loop works.