How do I build evals for an AI agent?

Install whileai (pip install whileai), run whileai init in your repo, then tell your coding agent to build better evals. It follows the strengthen-your-evals skill: find what your tests never reach, simulate a frozen held-out set, write the judge as a program and check it against people, report pass@1 with a 95% interval per behavior, measure the noise floor, and post every score to the Runs page. No GPU, no key until you report.

Does this work with Claude, GPT or Gemini agents?

Yes. The agent stays on the model you run. whileai measures it: one run per harness version (prompt, tools, model), each with its fingerprint, on the same frozen asks. A worked example is a refund agent on Claude Haiku 4.5.

How many test cases does an agent eval need?

Enough that the interval can resolve the gain you care about. whileai.holdout_size(effect, rows=) reads the spread off your own paired rows and says how many asks a gain needs; 64 asks resolve about 13 points, 5 points takes a few hundred. The Runs page shows the resolvable difference next to n on every behavior.

How do I know if my evals are any good?

The Evals table on the Runs page checks each behavior against the literature: a frozen named test, n at least 50 with the resolvable difference, judge agreement at least 0.8 on 50 human labels, length bias, a measured noise floor, contamination 0, reward not the judge, and a base that can still fail. Each cell says the rule, its source and the call that fixes it.

What does whileai init write?

A block under twenty lines in AGENTS.md (Codex, Cursor and Copilot read it), an @AGENTS.md include in CLAUDE.md (Claude Code reads it), and the tested skills under .claude/skills/. It then runs the evals skill's check.py so the agent sees the verdict line before it touches your code.

One prompt

Build better evals for your agent.

Paste this into Claude Code, Cursor or Codex. The agent installs whileai, follows the strengthen-your-evals skill, and hands you an eval that can actually fail. No GPU, no key.

Paste into your coding agent

Use whileai to build me better evals for my agent.

1. Install it: uv add whileai
2. Follow this skill: https://github.com/whilehq/whileai-sdk/blob/main/skills/strengthen-your-evals/SKILL.md
3. Read the docs at docs.withwhile.com when you need more.
4. You know my repo best: name the agent, behaviors and versions in its words. Rules: https://docs.withwhile.com/platform/naming

Or set the repo up once so every session finds the playbook without a URL:

pip install whileai
whileai init

Writes a short block in AGENTS.md, points CLAUDE.md at it, installs the tested skills under .claude/skills/, and runs the evals check.

What the agent does with it

01
Wrap the agent
Your tools are the functions you already have. The agent is any callable or an OpenAI-style client.
02
Write the judge as a program
A program checks the answer wherever one can. A model judge is used only where no program can, and it is checked against human labels first.
03
Simulate the situations
A covering array over tools, world state and user stance, with tool faults on a schedule. Offline, no key, a few hundred rows in a minute.
04
Count what can fail
A prompt the agent always passes measures nothing. The agent reports how many prompts can fail at all, then steers the mix toward them.
05
Size and report
pass@1 with a bootstrap interval over tasks, and how many prompts a real gain needs. Training is the step after this, when you want it.

Questions

How do I build evals for an AI agent?: Install whileai (pip install whileai), run whileai init in your repo, then tell your coding agent to build better evals. It follows the strengthen-your-evals skill: find what your tests never reach, simulate a frozen held-out set, write the judge as a program and check it against people, report pass@1 with a 95% interval per behavior, measure the noise floor, and post every score to the Runs page. No GPU, no key until you report.
Does this work with Claude, GPT or Gemini agents?: Yes. The agent stays on the model you run. whileai measures it: one run per harness version (prompt, tools, model), each with its fingerprint, on the same frozen asks. A worked example is a refund agent on Claude Haiku 4.5.
How many test cases does an agent eval need?: Enough that the interval can resolve the gain you care about. whileai.holdout_size(effect, rows=) reads the spread off your own paired rows and says how many asks a gain needs; 64 asks resolve about 13 points, 5 points takes a few hundred. The Runs page shows the resolvable difference next to n on every behavior.
How do I know if my evals are any good?: The Evals table on the Runs page checks each behavior against the literature: a frozen named test, n at least 50 with the resolvable difference, judge agreement at least 0.8 on 50 human labels, length bias, a measured noise floor, contamination 0, reward not the judge, and a base that can still fail. Each cell says the rule, its source and the call that fixes it.
What does whileai init write?: A block under twenty lines in AGENTS.md (Codex, Cursor and Copilot read it), an @AGENTS.md include in CLAUDE.md (Claude Code reads it), and the tested skills under .claude/skills/. It then runs the evals skill's check.py so the agent sees the verdict line before it touches your code.

Read the worked example

A refund agent on Claude Haiku 4.5, from a saturated eval to one that separates versions, with every number and its interval.

Evals that get harder The skill the agent follows Docs