We trained a support agent to hand customers off at the right time

The short version. If you fine-tune a support agent today, you probably pay people who know the domain to write example conversations. We wanted to know whether a simulator could do that job with nothing but the agent's tools and its policy document. So we let it write 1,057 conversations for a public support benchmark, trained the same model an expert-data vendor had trained, with their recipe, and scored both on the benchmark's own test. Our model solved 30% of the telecom tasks. The expert-trained model solved 17.5%, and the untrained model 5%. The gain over the untrained model is beyond doubt. The margin over the expert data rests on 40 tasks, so we call it promising rather than settled. Every training row, with the reason it was kept, is public on Hugging Face.

The experiment

tau2-bench is a public benchmark from Sierra. A model plays a support agent for an airline, a retailer or a telecom, with real tools and a policy document, while a second model plays the customer. A task passes only when the state of the world at the end matches the answer key, so the agent cannot pass by sounding right.

In April 2026 the expert-data company AfterQuery published a run on it. They paid domain experts to write 1,057 conversations, fine-tuned an 8-billion-parameter open model on them, and shared the recipe. That handed us a clean experiment: same base model, same recipe, same official scorer, and only one thing changed. Our 1,057 conversations came from a simulator that had seen the agent's tool schemas and policy document, and nothing else.

How the simulator writes a conversation

It maps the situations before it writes any. If you ask a model for a thousand customer requests, you get a thousand polite versions of the same request. So the simulator reads the policy document as data first. The telecom policy's 16 rules become one axis and the 13 tools a second, and four more axes describe the world: whether the tool works, what state the customer's account is in, how the customer behaves, and what happened before this conversation. Six axes describe hundreds of thousands of situations, far more than anyone can write. The simulator picks a set in which every pair of values appears together at least once, a standard trick from software testing called a pairwise covering design, because most real failures come from two things interacting rather than six. For telecom that came to 240 situations, and 150 of them were written out.

It gives each customer a way of typing. A second draw decides how the person writes: all lowercase, no punctuation, clipped, full of typos, or run-on. Of the 310 telecom conversations that trained the model, 126 are ordinary prose and the rest are not.

It plays the conversation out. One hosted small model plays both sides. As the agent it has the real tools and the policy. As the customer it has the situation card and nothing about the policy. A fake back-end answers every tool call the same way for the same seed, so a run can be repeated exactly.

It keeps or drops. Generating is cheap, so the training set is really made at this step. Rule-based checks first remove conversations that are structurally wrong: the agent claims an action it never took, uses an account number the customer never gave, or declares success after a failed tool call. Then a grader model reads every survivor against a written rubric and keeps or drops it with a reason. For telecom, 369 conversations went in and 310 came out. The reasons sit next to each row in the public dataset.

What the model learned

Twelve of the 40 telecom test tasks are ones where the right outcome is to hand the customer to a person. The trained model passed nine of them. A typical winning transcript goes like this: the phone number lookup fails, the name lookup returns nothing, it returns nothing again, and the model transfers the customer instead of guessing at an identity. Fifteen of the 310 training conversations contain that pattern, and nobody wrote them. The simulator drew them from the policy's own rules about when to escalate. The passes split evenly across the benchmark's hard and easy customer tags, so the gain is not confined to cooperative customers.

Three caveats belong next to that. The model learned when to hand off better than it learned when to keep going, so restraint is now a graded dimension in the next set. Retail and airline, trained the same way, did not match the expert-data numbers. And the expert-data figure is their best checkpoint per domain, while ours is the final checkpoint.

What this teaches about post-training

Post-training is where an agent's behavior is made. The base model already knew everything it needed. Forty-eight minutes on one GPU changed nothing about what it knows and everything about which behavior it reaches for: 5% to 30% on the same weights. The base model sets the ceiling, and the training conversations decide what the agent does under it.

The expensive part was never the writing. It was choosing which situations to write. An expert earns their fee by knowing that a suspended line plus a failed lookup plus an impatient customer is a case that matters. A policy document and a covering design found that case without being told. That is a measured claim, 30.0% against 17.5% on a 40-task test, which is exactly why the next run is the full 114-task split with repeated trials.

Grading is where a dataset gets its quality. Anyone can generate ten thousand conversations. The 310 that trained the model are the ones that survived a structural check and a rubric.

A number is only as good as its test. The test set is the benchmark's own held-out split, and every training row was checked for overlap against all 2,449 benchmark tasks before training. With that check, and the p-value above, 30% is a result rather than a story about memorization. These positions follow the RLHF book [5, §16].

For researchers

Single trial on the official test split, GPT-4.1 user simulator at temperature zero, protocol pre-registered and matched to AfterQuery's published run line by line [2]. Pass counts are $2$, $7$ and $12$ of $40$. On a two-sided Fisher exact test [6] over the $2 \times 2$ table of passes and failures, the simulated-data model separates from the untrained one, $p = 0.0064$, while the margin over the expert rows does not, $p = 0.29$. At 40 tasks that margin is suggestive rather than settled.

Situation space. Six axes: 16 policy clauses, 13 tools, tool condition (success, timeout, malformed, stale, permission denied), world state (exists, missing, already acted on, duplicate, partially completed), customer stance (ten values from ordinary to adversarial), and history (fresh, prior failure, prior partial action, contradicts earlier, repeat visit). Situations are drawn as a strength-2 covering array over the six axes [3]: a set of rows such that for every two axes with $v_a$ and $v_b$ values, each of the $v_a \times v_b$ value pairs appears in at least one row. That gives 240 regions. 150 were rendered, with an embedding-diversity objective choosing among candidates.

Persona sampling. Per row, a second layer draws texture, tone, length and pressure, with the split charted above. The writer model receives these as a prose aside, never as labels.

Rollouts. One hosted Qwen3-4B-Instruct-2507 plays both the agent (real tools, policy) and the user (situation card, no policy). A mock world answers every tool call, deterministic per seed. Two rollouts per situation, six-turn target, median eight messages.

Grading. Deterministic conduct checks (action claimed without a tool call, identifier the user never gave, success declared after a failed call), then an LLM grader against a written rubric with a keep-or-drop reason per row. Telecom pool 369, kept 310.

Recipe (AfterQuery's, unchanged):

The benchmark is τ²-bench [1]. The recipe is AfterQuery's [2], with the adapter as LoRA [4]. The rows, with each row's grader rule and reason, are the public dataset [7].

References

1. Barres, V., Dong, H., Ray, S., Si, X. and Narasimhan, K. (2025). τ²-Bench: Evaluating conversational agents in a dual-control environment. arXiv:2506.07982.
2. AfterQuery (2026). How AfterQuery expert data drives model performance on τ²-bench. afterquery.com. Accessed September 9, 2026.
3. Kuhn, D. R., Wallace, D. R. and Gallo, A. M. (2004). Software fault interactions and implications for software testing. IEEE Transactions on Software Engineering, 30(6), 418-421.
4. Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L. and Chen, W. (2022). LoRA: Low-rank adaptation of large language models. International Conference on Learning Representations. arXiv:2106.09685.
5. Lambert, N. (2025). Reinforcement Learning from Human Feedback. arXiv:2504.12501. Online at rlhfbook.com.
6. Fisher, R. A. (1935). The logic of inductive inference. Journal of the Royal Statistical Society, 98(1), 39-82.
7. Zero-Proof-AI (2026). tau2-simulated [dataset]. Hugging Face. huggingface.co/datasets/zero-proof-ai/tau2-simulated.
8. whilehq (2026). whileai SDK [software]. Apache 2.0. github.com/whilehq/whileai-sdk.

Run it

The simulator is the open-source whileai SDK. This is the call that produced the set, and it takes any agent's tools and system prompt:

pip install whileai

import whileai.simulations as wai
 
data = wai.simulate(
    agent="openai:gpt-4.1-mini", tools=TOOLS, system_prompt=POLICY
)
data.grade(
    rubric=RUBRIC
)  # the hosted judge writes reward and reason onto every row
print(data.pass_at)

The SDK README walks from that call to a gated training set in five steps.