Compare
While. vs Judgment Labs.
Judgment Labs finds the failure and measures how often it happens. While. learns from production traces and continuously improves the model with mid-training and post-training SFT and RL.
Why you need While.
Monitoring tells you the agent failed. It does not make the agent stop failing.
The fix is a change to the model. That takes three things: training data from the failures, a training run, and a held-out evaluation with an interval on the difference.
While. does all three. The other tools on this page stop at the red check.
What Judgment Labs is
Judgment Labs is tracing, judges and monitoring for agents in production. You send traces, search them for failures, write a judge that scores the behavior, track that behavior as a rate over time, and test a fix against saved cases before you ship it. It has an agentic judge for long multi-step runs, a Slack bot and an MCP server for investigating failures, and a feature that discovers new behaviors from unlabeled traffic.
The judgeval SDK is open source under Apache 2.0, in Python and TypeScript. The platform is closed, with self-hosting listed as coming soon. It raised $32 million in seed and Series A funding led by Lightspeed in May 2026. Its site names DoorDash, Monaco, Human Behavior, Contrario, E3 Group and Vigil Labs as customers.
What While. is
While. is a mid-training and post-training platform for language models, built for the team that owns a company's models. The open-source SDK builds and optimizes the dataset: it learns from production traces, writes situations from the agent's tools, plays the agent against them, and scores every run with a judge you write as a program. A hosted trainer runs mid-training and post-training on an open model with the latest SFT and RL algorithms, the model is served on your own endpoint, and a held-out set with a confidence interval says whether the change is real. Curb a behavior, build character, make tool use more efficient, harden the model against adversarial attacks.
The SDK is open source under Apache 2.0 and runs offline with no key. The platform is hosted. We do not publish customer names yet. Public results: a telecom support agent, trained to hand customers off at the right time, went from 5% to 30% on the held-out split of τ²-bench, and a refund agent, trained to refund only within policy, went from 18% to 85% on a held-out set.
Side by side
| Judgment Labs | While. | |
|---|---|---|
| What it does | Traces an agent in production, finds failures, scores them with judges, tracks behaviors over time, and tests a fix against saved cases. | Tests the agent, turns the failures into training data, trains a model, serves it, and measures the change on a held-out set with a confidence interval. |
| Open source | The judgeval SDK under Apache 2.0, Python and TypeScript. The platform is closed. Self-hosting is listed as coming soon. | The whileai SDK under Apache 2.0, Python only. The platform is closed. |
| Can you use it today | The SDK, yes. The platform needs a key from them. There is no self-serve signup. | Yes. pip install whileai. It runs offline with no key, and signup gives a trial key. |
| Judges | Judges written as natural-language rubrics or as Python. An agentic judge that searches a long run and checks its actions. | A judge is a program over the tool calls. Before you trust its scores, the SDK checks it against hand labels and probes it for length bias and other easy tricks. |
| Behaviors | A judge output tracked as a rate in production, with new ones discovered from unlabeled traffic. | A named behavior scored on every run, in simulation and in production, so the same number reads before training and after. |
| Regression tests | Offline tests: one judge version against one dataset version, baseline versus candidate. | A paired before and after with a 95% confidence interval on the difference. A run where no tool was called is refused, not reported. |
| Simulation | None. | Writes users and situations from the agent's tools and plays them, so you can test before you have traffic. |
| Training | None on its pages. Its writing says judge scores should become rewards for training. | SFT, DPO and GRPO on hosted GPUs. Adapters go to Hugging Face. The model is served on an OpenAI-compatible endpoint. |
| Investigation tools | Search across traces, a Slack bot, an MCP server, a CLI. | A traces page and a behavior board. No Slack bot, no MCP server. |
| Customers | DoorDash, Monaco, Human Behavior, Contrario, E3 Group, Vigil Labs. | Design partners in finance and customer support. Names not published yet. |
| Funding | $32 million, seed and Series A led by Lightspeed, May 2026. | Not disclosed. |
Pick Judgment Labs if
- You want to search and triage failures across a lot of production traffic.
- You want a Slack bot or an MCP server for investigating incidents.
- You write TypeScript.
Pick While. if
- You want the agent to get better, not only to know when it got worse.
- You have little or no traffic yet and still need a test.
- You want the test to run offline, with no key, in seconds.
- You want your own model on your own endpoint, with the weights.
Judgment Labs alternative
If you came here for a Judgment Labs alternative: While. covers judges, behaviors tracked as rates, and regression tests against saved cases, in an open SDK that runs offline with no key and a platform you can sign up for yourself. It does not match Judgment's trace search, Slack bot or TypeScript SDK. What it adds is simulation for teams that have no traffic yet, and training, so the failures become a better model.
Questions
- Is Judgment Labs open source?
- The judgeval SDK is open source under Apache 2.0. The Judgment platform, where traces and judge results go, is closed. Self-hosting is listed as coming soon.
- Does Judgment Labs train models?
- Not on its pages as of September 2026. It traces, judges and monitors. Its writing says judge scores should become rewards for training, but there is no training product yet.
- Can I use Judgment Labs and While. together?
- Yes. Both read OpenTelemetry traces. You can keep Judgment for monitoring and use While. to cut training data from the same runs and train on it.
Sources
Everything here about Judgment Labs comes from these pages, read on September 17, 2026. If something changed, tell us and we will fix it.