Jev, the new decision model from TypeSafe, as the judge for agent training data

The short version. On September 15, 2026, TypeSafe AI came out of stealth with Jev, which it calls a System One model [1]. You give it some state and typed questions, and it answers each with a probability, generating no tokens. The judge is where a training set gets its labels, so we added Jev as a judge to the whileai SDK. We have the plumbing and the tests, not a live run: Jev is on a waitlist and so are we. There is no agreement number here, and no claim of one.

Jev answers questions instead of writing text

Jev takes the rubric and the evidence as state, plus questions of three kinds, and answers them all in one pass [2]:

• A yes or no statement, answered as the probability it is true. TypeSafe calls this a noul. "The agent refunded without confirming the order" comes back as 0.93.
• A choice from labels you name, up to 255 of them, answered as the winning label plus the whole distribution. "Which team handles this ticket" comes back as billing, with billing: 0.81, technical: 0.12, account: 0.07.
• A score on an ordered rubric of two to ten levels, answered as the expected level. "How urgent is this, 0 to 2" comes back as 1.4.

The probabilities are the point. TypeSafe says they are calibrated: across many answers, the ones it gives a 0.9 to are right about nine times in ten [2]. That holds for the group, not any one answer, and it is what a judge needs. The vendor's other numbers, which we have not measured: 70 to 500 milliseconds end to end, $0.042 per million input tokens with output free, a 64,000-token cap on state plus questions, and 1,200 requests a minute [1, 3]. TypeSafe raised $40 million led by DCVC and is run by Diogo Almeida, formerly of OpenAI [4]. There are no open weights, no fine-tuning, and the model reads text only [2]. So Jev cannot be the model you train. It can grade the one you do.

A judge should give you a probability, not a sentence

We treat the judge as a measuring instrument: checked against people, never the model being trained, never the training reward and the test scorer at once. Chat judges prefer their own writing, the longer answer, and whichever answer they read first [5, 6, §7].

There is a quieter problem. A row the judge was sure about and a row it would flip on a re-roll both land in the training set as a 1. A probability fixes that in three ways:

1. A threshold you can state. A run passes when the probability that the agent did what it should is at least a half.
2. A band you can route. Between 0.4 and 0.6 the verdict is a coin. The SDK marks that row unsure, counts it, and leaves it for a person.
3. A failure class you do not have to guess. Today the SDK sorts a failed run into a class like fabrication by running regular expressions over the judge's sentence. With Jev, the class is a second question, and the answer is the class.

How to grade with Jev in the SDK

Same refund agent as our Raindrop post. Only the judge changes.

1. Get the runs. Simulate them, or load production traces. Each run is a JSON record: request, policy, tool calls with results, final reply.

2. Grade with Jev. One spec string picks the judge. The key comes from TYPESAFE_API_KEY.

import whileai.simulations as wai
 
# the refund agent from the Raindrop post
from agent import TOOLS, POLICY, answer
 
data = wai.simulate(
    answer, tools=TOOLS, system_prompt=POLICY, situations=40, repeats=4
)
report = data.grade(
    rubric="Refund delivered orders within 30 days, after looking the order up.",
    spec="typesafe:jev-latest",
)
 
print(report["n0"], report["n1"], report["unsure"])
for row in data.trajectories[:3]:
    print(
        row["reward"], row["judge_meta"]["confidence"], row.get("failure_class")
    )

Each run is one request to Jev. The rubric goes in as instructions, the run as record, and two questions ride along. This is the exact body, from the SDK's tests:

{
  "model": "jev-latest",
  "state": {
    "instructions": "Grade the agent against this rubric. Refund delivered orders within 30 days, after looking the order up. Score 1 only if the agent did the task the way the rubric requires and claimed nothing the tools did not return ...",
    "record": {
      "tools": ["lookup_order", "issue_refund"],
      "situation": "I want a refund on A1002",
      "final_text": "Refunded $250 to your card.",
      "agent_policy": "Refund delivered orders within 30 days. Always look the order up first.",
      "steps": [{"tool": "lookup_order", "arguments": {"order_id": "A1002"}, "result": "total $250, delivered 5 days ago"}]
    }
  },
  "questions": {
    "verdict": {
      "type": "noul",
      "instructions": "The agent did what it should have done. Judge only what the record shows ...",
      "criteria": {"true": "The agent completed the request the way the instructions require ...", "false": "The agent claimed a result the tools did not return ..."}
    },
    "failure_class": {
      "type": "choice",
      "instructions": "If the agent failed, the kind of failure that fits best; 'none' when the agent did what it should.",
      "criteria": {"none": "...", "fabrication": "...", "unconfirmed_write": "...", "junk_output": "...", "fault_dishonesty": "...", "arithmetic": "...", "no_attempt": "...", "incompleteness": "..."}
    }
  }
}

3. Read what came back. The reward is still 0 or 1, so pass@1, the paired before-and-after and the trainer export work unchanged. New on each row: judge_meta.confidence, and judge_meta.unsure on the coin flips. On a failed row, failure_class is what Jev chose.

4. Check the judge against people. Label a sample by hand, attach the labels, ask for agreement:

wai.attach_labels(data.trajectories, "labels.jsonl", kind="human")
print(wai.judge_trust(data.trajectories))  # agreement and Cohen's kappa

We have not run step 2 or step 4 against the live service. The tests hit a fake server that answers in the shape TypeSafe's own SDK documents [7]. When our key arrives, the first thing we publish is that agreement number with its interval, beside the same number for the hosted chat judge on the same rows. Until then, "Jev is a better judge" is not proven.

The same spec works for the pairwise judge (one choice: A, B or tie), the rubric judge (one yes-or-no per item) and the audit, which asks the verdict blind. Where a chat model is needed, the agent, the writer, the simulated user, the spec is refused. Jev can judge a conversation. It cannot have one.

What this teaches about post-training

Post-training is where an agent's behavior is made, and the judge is where the training set gets its labels. A judge that is wrong one time in five teaches the model the wrong thing one time in five [6, §7]. A calibrated probability is not a better verdict. It is the same verdict with its uncertainty attached, so you can spend human attention where the instrument is unsure and nowhere else.

Jev cannot grade its own writing because it does not write, so the judge and the policy stay apart by construction.

Price changes what you grade. At the vendor's numbers, ten thousand runs of about two thousand tokens each is roughly twenty million input tokens, under a dollar. That is arithmetic on their price sheet, not a measurement. If it holds, grading every rollout at every RL step stops being a budget line.

For researchers

Decision rule. For a run $x$ with rendered record $r(x)$ and judge prompt $s$, the state is $(s', r(x))$, where $s'$ is $s$ minus its reply-format sentences, plus two questions. The verdict question returns $p = P(\text{did what it should} \mid s', r(x))$ and

with $c$ stored as judge_meta.confidence. A row is unsure when $|p - 0.5| < 0.1$, a width we have not tuned (DECISION_UNSURE_BAND). The failure-class question is a choice over the eight labels in preflight.FAILURE_CLASSES plus none, and its argmax is kept only when $\hat{y} = 0$.

Judge identity. judge_version is <model>@sha256(s' || Q)[:12], $Q$ the serialized questions, so a prompt or question edit is a new judge.

Checks to run with a key. Agreement and Cohen's $\kappa$ [8] against human gold via judge_trust, bootstrap interval over rows. A reliability curve of $p$ in ten bins and the expected calibration error [9]. The length-bias probe in judge_probes. All three against the hosted chat judge on the same rows.

Transport. POST /v1/systemone over requests, retried on 5xx and 429 with retry-after honored, 30-second timeout. Warm-up is GET /v1/models, so a bad key fails once. The record is capped at 8,000 characters, well under Jev's 64,000-token limit.

Code. whileai/simulations/generate/typesafe_backend.py (wire) and whileai/simulations/score/decision_judge.py (questions and verdicts), merged in whilehq/whileai-sdk#413 with 45 offline tests against the shapes in typesafe-sdk 0.7.0 [7].

References

1. TypeSafe AI (2026). Introducing System One Models and Jev. typesafe.ai/blog/introducing-system-one-models-and-jev. Accessed September 18, 2026.
2. TypeSafe AI (2026). System One: concepts. docs.typesafe.ai/concepts/system-one. Accessed September 18, 2026.
3. Valyu (2026). How to use Jev: a practical guide to TypeSafe's System One model. dev.to. Accessed September 18, 2026.
4. heise online (2026). AI model "Jev" to make machines decide faster. heise.de. Accessed September 18, 2026.
5. Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., et al. (2023). Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. arXiv:2306.05685.
6. Lambert, N. (2025). Reinforcement Learning from Human Feedback. arXiv:2504.12501. Online at rlhfbook.com.
7. TypeSafe AI (2026). typesafe-sdk 0.7.0 [software]. pypi.org/project/typesafe-sdk.
8. Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37-46.
9. Guo, C., Pleiss, G., Sun, Y., and Weinberger, K. Q. (2017). On calibration of modern neural networks. ICML. arXiv:1706.04599.
10. whilehq (2026). whileai SDK [software]. Apache 2.0. github.com/whilehq/whileai-sdk.

Run it

uv add "whileai @ git+https://github.com/whilehq/whileai-sdk"
# or: pip install "whileai @ git+https://github.com/whilehq/whileai-sdk"
export TYPESAFE_API_KEY=...   # console.typesafe.ai, after the waitlist

The Jev judge is on main and ships in the next whileai release. With no spec=, the same call uses the hosted chat judge.

FAQ

What is Jev? A model from TypeSafe AI, released September 15, 2026, that answers typed questions about text or JSON with a probability on each answer, and writes no text. TypeSafe calls the category a System One model.

What is a System One model? A model built for fast, structured decisions: a yes or no, a choice from labels you name, or a score on a rubric, each with a calibrated probability.

How much does Jev cost and how fast is it? Per TypeSafe, $0.042 per million input tokens with output free, and 70 to 500 milliseconds per request. We have not measured either.

Can Jev replace an LLM judge? For classification, scoring and pass-or-fail grading, yes, with a probability the chat judge does not give. It cannot write a reason or play the agent. Whether it agrees with humans more often than a chat judge is a number we have not measured.

Can you fine-tune Jev or run it locally? No. As of September 2026 there are no open weights and no fine-tuning, and access is through the API behind a waitlist.

How do I use Jev with the whileai SDK? Set TYPESAFE_API_KEY and pass spec="typesafe:jev-latest" to grade, the pairwise judge, the rubric judge or the audit. It is refused for the agent, the writer and the simulated user.