An open-source alternative to Raindrop Simulations

The short version. On September 17, 2026, Raindrop announced a Series A and a product called Simulations. Per the product page, it replays your production traffic against a proposed change, runs anomaly detection on the results, and can block the pull request. It is hosted, closed source, and behind a waitlist. If you want that test today, the whileai SDK is open source and runs it. One script runs main and the pull request on the same real requests, four times each, exits 1 when something got worse, and hands you the failed runs to train on.

The same test on the SDK

Here is that agent, built on the Anthropic SDK's tool runner. The only thing we added for testing is the return value: the tool calls it made and the reply it wrote.

# agent.py
import anthropic
from anthropic import beta_tool
 
ORDERS = {  # total, days since delivery
    "A1001": (80, 12),
    "A1002": (250, 5),
    "A1003": (40, 20),
    "A1004": (120, 9),
}
 
POLICY = (
    "Refund delivered orders within 30 days. Always look the order up first."
)
 
 
@beta_tool
def lookup_order(order_id: str) -> str:
    """Look up an order by id. Orders on file: A1001, A1002, A1003, A1004.
 
    Args:
        order_id: The order id, like A1001.
    """
    if order_id not in ORDERS:
        return "not found"
    total, days = ORDERS[order_id]
    return f"total ${total}, delivered {days} days ago"
 
 
@beta_tool
def issue_refund(order_id: str, amount: float) -> str:
    """Refund an order.
 
    Args:
        order_id: The order id.
        amount: Dollars to refund.
    """
    return "ok"
 
 
client = anthropic.Anthropic()
 
 
def answer(message: str, policy: str = POLICY) -> dict:
    runner = client.beta.messages.tool_runner(
        model="claude-opus-5",
        max_tokens=1024,
        system=policy,
        tools=[lookup_order, issue_refund],
        messages=[{"role": "user", "content": message}],
    )
    steps, reply = [], ""
    for turn in runner:  # one turn per model call
        for block in turn.content:
            if block.type == "tool_use":
                steps.append({"tool": block.name, "arguments": block.input})
            elif block.type == "text":
                reply = block.text
    return {"steps": steps, "final_text": reply}

The test is a second script. The judge is a program that reads the tool calls rather than the reply, so an agent that says it refunded without calling issue_refund fails.

# ci_check.py
import sys
 
import whileai.simulations as wai
 
from agent import ORDERS, POLICY, answer, issue_refund, lookup_order
 
# The pull request changes one line of the policy.
POLICY_PR = POLICY + " Orders over $200 need a manager: do not refund."
 
 
def main_agent(message):
    return answer(message, POLICY)
 
 
def candidate_agent(message):
    return answer(message, POLICY_PR)
 
 
def judge(row):  # the policy as a program, read off the tool calls
    steps = row.get("steps") or []
    looked = [s for s in steps if s["tool"] == "lookup_order"]
    refunded = any(s["tool"] == "issue_refund" for s in steps)
    oid = looked[0]["arguments"]["order_id"] if looked else None
    allowed = oid in ORDERS and ORDERS[oid][0] <= 200
    return {
        "reward": float(bool(looked) and refunded == allowed),
        "markers": {
            "looked_up_first": float(bool(looked)),
            "refund_only_when_allowed": float(refunded == allowed),
        },
    }
 
 
traces = wai.load_traces("traces.jsonl")  # yesterday's traffic
asks = [t["prompt"] for t in traces]
common = dict(
    tools=[lookup_order.to_dict(), issue_refund.to_dict()],
    system_prompt=POLICY,
    seeds=asks,
    situations=len(asks),
    mode="rl",
    repeats=4,
    repeat_policy="fixed",
    simulator=False,  # replay the asks word for word, no whileai key
    reproducible=True,
    concurrency=4,
)
 
before = wai.evaluate(wai.simulate(main_agent, **common), judge)
after = wai.evaluate(wai.simulate(candidate_agent, **common), judge)
print(wai.pass_at(before.rows))
print(wai.pass_at(after.rows))
 
report = wai.delta_report(
    before.rows,
    after.rows,
    target="pass_at_1",
    must_not_regress=["looked_up_first", "refund_only_when_allowed"],
)
print(report["target_verdict"], report["target_delta"], report["target_ci95"])
for note in report["warnings"]:
    print("!", note)
 
# A guarded regression fails the pull request.
sys.exit(0 if report["ok"] else 1)

The requests are the JSONL your tracing tool already writes, one per line.

{"ask": "Refund A1002, the laptop arrived cracked.", "steps": [], "final_text": ""}

Every request runs four times per version. pass@1 gets a 95% confidence interval by bootstrap over requests, and so does the paired before-and-after difference. You name the behaviors that may not get worse, and if one of them scored the same on every run on both sides, the report tells you that check cannot fail.

A run on a small refund agent

We ran the two files above with Claude Opus 5 on eight requests about four orders. The pull request adds one sentence to the policy: orders over 200 dollars need a manager.

The report also says how many requests it would take to prove a gain of that size. Run it the other way and the difference is minus 25 with the mirrored interval: eight requests cannot prove a 25 point regression either, and the tool says so instead of showing you a green check.

Where the two differ

Raindrop does two things we do not: service copies built from your schemas, and a GitHub integration that blocks the merge. If you need those and can wait, Raindrop is the better choice. The full comparison is on While vs Raindrop.

Why we built it this way

Test on your own traffic, because public benchmarks are in every model's training data and your customers' requests are not.

Put an interval on the difference. The same test on the same agent moves a few points between runs, and without an interval you cannot tell a regression from noise. The RLHF book [1, §16] says it plainly: a result needs a held-out set and a confidence interval, or it is not a result.

Keep the failed runs. The requests an agent passes sometimes are the ones reinforcement learning can improve. The book puts that band at 20 to 80 percent, and the SDK keeps it by default. A regression tool throws those runs away.

Use one judge for testing and for training, because two judges drift apart and production is where you find out.

For researchers

Each replayed request $i$ gets $n = 4$ rollouts (repeats=4, repeat_policy="fixed") and $c_i$ of them pass. pass@$k$ is the unbiased estimator of Chen et al. [2], pass$^k$ the share of tasks that pass on every sample, and the 95% intervals are percentile bootstraps over requests [3] with $B = 2{,}000$ resamples:

A before-and-after comparison is paired on the same requests. With $\hat p_i = c_i / n$ on each side, the reported difference and its interval come from bootstrapping the per-request differences:

delta_report runs that bootstrap with seed 0 and applies the same test to every marker named in must_not_regress. Its verdicts are improved, moved_the_wrong_way, no_change_detected, within_eval_noise when run_std or eval_variance supplies the re-run noise, and moved_unreplicated when the move clears the interval but no re-run was given. The number of requests needed to resolve a given gain is the power calculation in An eval that gets harder as the agent improves.

Offline, seeds are replayed word for word when situations equals the number of seeds; simulate_from_traces writes new situations instead. wai.train(dataset, method=...) runs "sft", "dpo" or "grpo", and wai.serve(name, run) hosts the result. Our simulate-then-train result (5.0% to 30.0% on tau2-bench telecom) is in this post. Everything in this post about Raindrop comes from their announcement [4] and product page [5] as read on September 17, 2026.

References

1. Lambert, N. (2025). Reinforcement Learning from Human Feedback. arXiv:2504.12501. Online at rlhfbook.com.
2. Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. de O., Kaplan, J., et al. (2021). Evaluating large language models trained on code. arXiv:2107.03374.
3. Efron, B. (1979). Bootstrap methods: Another look at the jackknife. The Annals of Statistics, 7(1), 1-26.
4. Raindrop (2026). Announcing our Series A. raindrop.ai/blog/series-a. Accessed September 17, 2026.
5. Raindrop (2026). Simulations. raindrop.ai/simulate. Accessed September 17, 2026.
6. whilehq (2026). whileai SDK [software]. Apache 2.0. github.com/whilehq/whileai-sdk.

Run it

pip install whileai anthropic
python ci_check.py   # exit 1 fails the pull request

Tested on whileai 0.61 and anthropic 0.122. The SDK needs no key of its own; the agent needs ANTHROPIC_API_KEY, as it does in production. Drop simulator=False and run whileai login if you want the hosted writer to generate new situations from your traffic. See recipes/02-measure/eval-your-agent and the Evals docs.

FAQ

Is Raindrop Simulations open to use? Not as of September 17, 2026. It is waitlisted, with general availability over the next month.

What is an alternative to Raindrop Simulations? The whileai SDK: open source, the same replay test with a 95% confidence interval, exit 1 in CI, then training on the failures.

Can it run on every pull request? Yes, as a CI step that exits 1 on a failure. There is no app to install.

Does it need an API key? The SDK does not. Your agent needs whatever key it uses in production.

Does my agent have to change? No. Wrap it in a function that returns the tool calls it made and the reply it wrote.