Guide / September 21, 2026
Harness vs model
An agent is a model plus a harness: the prompt, the tools, the context files, the turn cap. On a closed model the harness is the only part you can change, and it often moves the score more than switching models. whileai now runs, versions and compares harnesses, and tells you which lever moved the score.
The short version. We made the harness a first-class object in the SDK. You build one, run it on frozen tasks, and every row remembers which harness produced it. A report then says whether the harness or the model moved the score, with the range the true answer very likely sits in. The caveat: a difference only counts once that range clears zero, and on a small task set it usually does not.
The harness is the lever you actually hold
A model never meets a task alone. Something builds its context, hands it tools, decides when to stop, retries, compacts, delegates. That program is the harness. If your agent runs on Claude Code, Codex, pi, or your own prompt-and-tools loop around a closed model, the harness is the whole thing you own.
It matters more than people expect. Among comparable frontier models, the harness explains more of the score spread than the model does, and can flip which model ranks first [1]. A harness found by search, with earlier candidates' scores and traces in view, beat the hand-built ones on a terminal benchmark [2]. And a policy trained under one fixed harness fell apart when the tools shifted [3].
One object, run like an agent, versioned like weights
A harness is a model, instructions and tools. Its version is a fingerprint
over everything a comparison has to disclose [1]: the model, the prompt,
the tool names, the context files it loads, the turn cap, how it compacts,
whether it retries or delegates. Edit the prompt or drop a new CLAUDE.md
in the folder and it is a new version without anyone naming it.
import whileai as wai
careful = wai.Harness("openai:gpt-4.1-mini", instructions=POLICY, tools=TOOLS, label="careful@mini")
coder = wai.Harness.claude_code("sonnet", cwd="repo/", max_turns=8)
data = wai.simulate(careful, tasks=frozen, mode="rl", repeats=4) # rows carry the harnessThe coding-agent presets run one subprocess per task and read the CLI's JSON stream into the same trajectory shape every other agent returns.
Which lever moved the score
Run every harness on every model over the same tasks, grade them with one judge, and hand the rows to one call. It splits the spread of the cell means into a harness part, a model part and their interaction, and puts a band on each by resampling tasks [4]. It also says whether the leading model changes with the harness.
A worked example on rows built by hand so it runs offline. Changing the harness moves the score by 20 points; changing the model by 5.
| Item | Value |
|---|---|
| careful on Claude Haiku 4.5 | 75 |
| careful on GPT-4.1 mini | 75 |
| eager on Claude Haiku 4.5 | 60 |
| eager on GPT-4.1 mini | 50 |
The report reads: the harness moved the score more than the model did. The band on the harness share, 61 to 97, is the range the true share very likely sits in.
| Item | Value | 95% interval |
|---|---|---|
| Harness | 89% | 61% to 97% |
| Model | 6% | 2% to 19% |
| Interaction | 6% |
The band is the result. The same rows cut to 40 tasks give the same shares with a harness band of 2 to 100, and the report says the difference could be chance instead of rounding it up. Forty binary outcomes cannot separate a 20-point harness effect from noise. Size the task set before you argue about harnesses.
Search it, then gate it
Once the harness is an object, optimizing it is a loop, not a vibe. The Meta-Harness recipe keeps every candidate as a file, shows the proposer (a coding agent) every prior candidate's source, score and worst rows, and only keeps a pick that beats the baseline on held-out tasks and on a held-out model [2]. The exported training environment can roll out under several harnesses, so an open model you train does not overfit to one [3].
For researchers
| What | Where |
|---|---|
| Fingerprint fields | model, instructions, sorted tool names, and the disclosure: context files, turn cap, compaction, retries, subagents, sampling [1] |
| Attribution | two-way decomposition of cell means over shared tasks; shares of total sum of squares; percentile bootstrap over tasks, 2000 draws, 95% [4] |
| Ranking reversal | the top model per harness; a flip is reported as a sentence, not a flag [1] |
| Live tested | the Claude Code preset, 2026-09-21; the Codex and pi parsers are checked on recorded stream shapes only |
| Not measured yet | a live Meta-Harness replication; the recipe's dry run shows the loop and claims no number |
Run it
The page with every block runnable offline is
docs.withwhile.com/reference/harness.
The search loop is
recipes/papers/meta-harness.
References
- Zhang, Wang, Ge, Xu, Hamm, Reddy. Stop Comparing LLM Agents Without Disclosing the Harness. 2026. arXiv:2605.23950
- Lee, Nair, Zhang, Lee, Khattab, Finn. Meta-Harness: End-to-End Optimization of Model Harnesses. 2026. arXiv:2603.28052
- Kim, Choi, Lee, Jun, Kim, Park. The Interplay of Harness Design and Post-Training in LLM Agents. 2026. arXiv:2606.25447
- Miller. Adding Error Bars to Evals. 2024. arXiv:2411.00640