Compare
While. vs Harbor.
Harbor scores agents on tasks someone wrote. While. learns from production traces and continuously improves the model with mid-training and post-training SFT and RL.
Why you need While.
Monitoring tells you the agent failed. It does not make the agent stop failing.
The fix is a change to the model. That takes three things: training data from the failures, a training run, and a held-out evaluation with an interval on the difference.
While. does all three. The other tools on this page stop at the red check.
What Harbor is
Harbor is the open-source harness behind Terminal-Bench. A task is a folder with an instruction, a Docker environment and a verifier that writes a reward. Harbor starts a sandbox per trial, runs the agent inside it, runs the verifier, and averages the rewards. It ships with 41 agents built in, Claude Code, Codex, OpenHands, Gemini CLI and Cursor among them, and can run thousands of trials in parallel on about 25 sandbox clouds.
Around it sits Harbor Hub, where teams publish datasets, share job results and post leaderboards, and a hosted service for remote runs that is behind an access form. Rewardkit gives verifiers reusable criteria and LLM or agent judges. A simulated user can play an existing task in conversation. Harbor is Apache 2.0, made by the Terminal-Bench team. Its pages name no company, no customers and no funding.
What While. is
While. is a mid-training and post-training platform for language models, built for the team that owns a company's models. The open-source SDK builds and optimizes the dataset: it learns from production traces, writes situations from the agent's tools, plays the agent against them, and scores every run with a judge you write as a program. A hosted trainer runs mid-training and post-training on an open model with the latest SFT and RL algorithms, the model is served on your own endpoint, and a held-out set with a confidence interval says whether the change is real. Curb a behavior, build character, make tool use more efficient, harden the model against adversarial attacks.
The SDK is open source under Apache 2.0 and runs offline with no key. The platform is hosted. We do not publish customer names yet. Public results: a telecom support agent, trained to hand customers off at the right time, went from 5% to 30% on the held-out split of τ²-bench, and a refund agent, trained to refund only within policy, went from 18% to 85% on a held-out set.
Side by side
| Harbor | While. | |
|---|---|---|
| What it does | Runs an agent on a set of tasks in sandboxes and reports the mean reward. Built to compare agents and models on a shared benchmark. | Tests the agent, turns the failures into training data, trains a model, serves it, and measures the change on a held-out set with a confidence interval. |
| Open source | The framework, under Apache 2.0. The Hub and hosted runs are a service. | The whileai SDK under Apache 2.0. The platform is closed. |
| Can you use it today | The framework, yes, with Docker. Hosted runs need an access form. | Yes. pip install whileai. It runs offline with no key. |
| Whose agent | Any of 41 built-in agents, any agent on the ACP registry, or your own. Made to compare harnesses. | Your own agent, as a Python function. Not made to compare harnesses. |
| Where the test comes from | A task you write: instruction, Docker image, verifier. Or a published dataset from the Hub, such as Terminal-Bench. | Your production traffic, or situations the SDK writes from your agent's tools when you have no traffic yet. |
| What the agent runs against | A real container, on your machine or on one of about 25 sandbox clouds. Thousands at once. | Fake tool backends built from your tool descriptions, or your real tools inside one function. |
| Judges | A verifier script per task. Rewardkit adds 20 or so built-in criteria and LLM or agent judges. | A judge is a program over the tool calls. Before you trust its scores, the SDK checks it against hand labels and probes it for length bias and other easy tricks. |
| Simulation | A simulated user with a persona talks an existing task through with the agent. It does not write tasks. | Writes users and situations from the agent's tools and plays them, so you can test before you have traffic. |
| The number you get | Mean reward across tasks, and a leaderboard on the Hub. No interval unless you write a metric. | A first-try pass rate with a 95% confidence interval, a rate per named behavior, and a before and after with an interval on the difference. |
| After the test | Rollouts you can take to an RL trainer. Harbor does not train, serve or watch production. | The failed runs become training data. Train with SFT, DPO or GRPO and host the result on your own endpoint. |
| Who is behind it | The Terminal-Bench team. No company, customers or funding on its pages. | Design partners in finance and customer support. Names not published yet. Funding not disclosed. |
Pick Harbor if
- You want to compare Claude Code, Codex and other agents on the same benchmark.
- Your tasks need a real container, and you want thousands of them at once.
- You are publishing a benchmark and want a leaderboard.
Pick While. if
- You own one agent and want it to get better on its own traffic.
- You have little or no traffic yet and still need a test.
- You want a pass rate with an interval, not a mean.
- You want your own model on your own endpoint, with the weights.
Harbor alternative
If you came here for a Harbor alternative: While. covers the part where an agent is run against a test and scored by a judge, but it starts from your agent's own traffic or from situations it writes, not from a task folder. It does not run 41 agents in 25 clouds, and it is not a benchmark harness. What it adds is everything after the score. The failed runs become training data, the model is trained and served, and a held-out set with a confidence interval says whether the change was real.
Questions
- Is Harbor open source?
- Yes. The framework is Apache 2.0 on GitHub. Harbor Hub and hosted runs are a service, and hosted runs are behind an access form.
- Does Harbor train models?
- No. Its pages say it can generate rollouts for RL, which you take to a trainer yourself. There is no training, serving or production monitoring in Harbor.
- Can I use Harbor and While. together?
- Yes. Harbor can write its trajectories as OpenTelemetry spans through a plugin, and While. reads OpenTelemetry. Run the benchmark in Harbor, then bring the failed trials into While. and cut training data from them.
Sources
Everything here about Harbor comes from these pages, read on September 17, 2026. If something changed, tell us and we will fix it.