Evals for agents on frontier models
Your evals saturate. Find the failures nobody wrote a test for.
Keep the model you run. While. writes the situations your agent has never seen, plays it against them, and reports pass@1 with a confidence interval. Offline, no key, ten minutes.
uv add whileaiimport whileai.simulations as wai
from agent import POLICY, TOOLS, answer, judge # your agent, your judge
common = dict(tools=TOOLS, system_prompt=POLICY, situations=40, repeats=4, seed=0)
probe = wai.evaluate(wai.simulate(answer, **common), judge)
print(wai.pass_at(probe.rows)) # pass@1 with a 95% interval
by_prompt = {}
for r in probe.rows:
by_prompt.setdefault(r["scenario_id"], []).append(r["reward"])
fails = [p for p, rs in by_prompt.items() if min(rs) < 1]
print(f"failure-capable: {len(fails)}/{len(by_prompt)}") # the ceiling on any gainOpen source under Apache 2.0, on GitHub. The first run needs no key: customers come from templates. Hosted runs add a writer model for variety.
What you get
- Situations, not prompts
- The writer reads your agent's tools and policy and builds a world that breaks on purpose: a tool that times out, a customer who will not cooperate, a request just outside the rule. Near-duplicates are dropped.
- pass@1 with a 95% interval
- Four attempts per situation. A rate per situation type, so a gain in one cannot hide a loss in another. A mean without an interval is not a result.
- Failure-capable prompts
- The count of situations your agent fails at least once. It is the ceiling on any gain a change can show. When it hits zero the eval has saturated and the writer is steered toward what still fails.
- A delta that exits 1
- Main against candidate on the same situations, a paired difference with its interval, and named behaviors that must not regress. It runs on every pull request.
On every pull request
before = wai.evaluate(wai.simulate(main_agent, **common), judge)
after = wai.evaluate(wai.simulate(candidate_agent, **common), judge)
report = wai.delta_report(
before.rows, after.rows,
target="pass_at_1",
must_not_regress=["looked_up_first", "refund_only_when_allowed"],
)
sys.exit(0 if report["ok"] else 1) # a guarded regression fails the pull requestWhat it found
A refund agent on Claude Haiku 4.5 passed 95% of the default situation mix, and only 2 of 40 situations could fail. Three steered sets later, 16 of 40 could, and the eval had found a real gap: 37 of 160 runs never called a tool.
Same agent, same model. The eval got harder as it was steered toward what the agent fails.
| Item | Value | 95% interval |
|---|---|---|
| Default mix | 95% | 88% to 100% |
| Steered: ambiguous and unsure customers | 91% | 82% to 97% |
| Steered: refund tool, the 30-day and $200 rules | 78% | 64% to 89% |
| Seeded with eight asks naming real orders | 70% | 56% to 82% |
When you are ready
The same scored runs are post-training data. While. cuts them into SFT rows, DPO pairs and GRPO groups, trains an open model on your own GPU or ours, and scores it on the held-out set you already have.
How the loop works