We trained an agent to write SQL that runs

The short version. We taught two small models to write SQL against a shop database, with reinforcement learning and no judge model in the loop. The reward is a program: it runs the query and compares the rows to a reference. One model, Nemotron-Nano-8B, went from 26% to 35% on a held-out test in a single 44-minute round, and the gain held up across three repeat scores. The other, Qwen3-4B, moved 3 points in four rounds, unproven. A fifth round with the same reward fed differently, bigger updates, only the difficulty band, cut-off replies ignored, took it from 53% to 74% on 459 held-out tasks. The recipe is public.

The recipe in six steps

Seed. Start with a Postgres database that has data in it. Ours has eight tables and 300 orders. Swap in yours.

Author. A teacher model writes each question together with its SQL. We run the SQL twice and keep it only if it returns one to fifty rows both times, then drop duplicates. That gave us 741 tasks. Before doing anything else, hold out the test set: 140 tasks chosen by a hash of the id, never trained on. Every number in this post comes from those 140.

Verify. The verifier runs the model's query read-only with a timeout and compares rows to the reference as a multiset, floats to two decimals, column order free, row order only when the reference has ORDER BY. A match scores 1.0, a query that runs but is wrong 0.1, and one that does not run 0. There is no judge, so a longer answer cannot score higher.

Benchmark. Sample each test task four times at temperature 0.7 and report pass@1 with a 95% interval, with pass@4 beside it. The gap between the two is what RL has to work with. Then read the failures, because they tell you what the reward will teach. Nemotron-Nano-8B could not run 55% of its own queries, and a verifier is very good at teaching that.

Train. GRPO with a LoRA of rank 16 and eight samples per prompt. vLLM generates inside the trainer and Postgres runs inside the container. 600 steps took 44 minutes on one H100. Leave gradient checkpointing off with TRL 0.19: on Qwen3-4B it produced garbage completions at reward zero.

Score. Serve the adapter, sample the test set four times, and take the paired difference against the base with its interval. We call a gain proven only when the interval excludes zero. A single evaluation run is marked unreplicated, so score both sides three times. Then start the next round from the best checkpoint, or stop.

What happened

Nemotron-Nano-8B moved in one round: 26% to 35%, a gain of 9 points with an interval of +5 to +13. The three repeat scores came in at +9, +10 and +11, all clear of zero. The share of its queries that failed to run fell from 55% to 35%, which is most of the gain.

Qwen3-4B was a different story. It started at 58% and we ran four GRPO rounds plus one SFT pass on its own verified outputs. The best checkpoint reached 61%, a gain of 3 points with an interval of -2 to +7, so we cannot say it improved. By round four the training reward sat at 0.65 and the KL to the base at 0.03. The model had stopped moving, and so did we.

Same recipe, two outcomes. The base that could not run its SQL moved in under an hour. The base that already could, and got the meaning wrong instead, did not. That is why we say read the failures before you buy training steps. In the first two Qwen rounds the training reward rose while the test score did not, and without the held-out set that run would have shipped.

Getting the first merged recipe to round four took fourteen hours, six checkpoints, twelve test-set scores and under $100 of GPU.

Round five: the same reward, fed differently

We compared the Qwen run with a larger one on another schema. The reward, the tasks and the model were the same. Three things about how the trainer was fed were not.

1. Answers per update. Rounds one to four updated after every question, eight answers at a time, a random walk. Round five looks at 32 questions times 16 answers, 512 per update.
2. Which questions. About half the questions had become always right or always wrong, and those teach nothing. Round five trains only on the 326 questions in the difficulty band, the ones the model got right some of the time on its own attempts.
3. Cut-off replies. A reply that ran out of room used to score zero, so the cheapest lesson was to think less. Round five ignores those replies instead of punishing them.

We also grew the test. 140 questions gave a band of about six points, so a real three-point gain could never be proven. 1,482 more questions, written the same way, took the held-out set to 459 and the band to about three and a half points. On that test the base scores 53% (the new questions are harder), round four 55%, and the round-four gain of +2.6 sits just outside the band. It had been real all along.

Step 50 was measured twice from scratch, 0.73 both times, and the base was scored through the same server as the checkpoints (+0.4 points, inside the band). Fifty steps took about seven hours on one H100 and 25,600 scored answers. Rounds one to four had used 32,000 answers over 8,000 steps for three points.

Read the other columns before calling it new capability. pass@4 barely moved, 76% to 79%. pass^4, right on every try, went from 25% to 65%. The model learned to produce the query it could already find sometimes, every time, and to stop before the reply budget: no-query replies went from 13% to none and SQL errors from 12% to 5%. That is what a verifier reward does on a difficulty band, and it is what reliable in production means.

Next. The curve is flat from step 50 and the pass@4 ceiling is flat at 79%. Re-measure the difficulty band on the round-five model so the questions it now always gets right drop out, then a bigger base or harder questions it can solve sometimes.

For researchers

Each held-out task $i$ gets $n = 4$ samples at temperature 0.7 and $c_i$ of them pass. pass@$k$ is the unbiased estimator of Chen et al. [3], pass$^k$ the share of tasks that pass on every sample, and the 95% intervals are percentile bootstraps over tasks [4] with $B = 2{,}000$ resamples:

A before-and-after comparison is paired on the same tasks. With $\hat p_i = c_i / n$ on each side, the reported difference and its interval come from bootstrapping the per-task differences:

Here $N = 140$. The verifier is a program, so the reward is verifiable in the sense of [1, §7]. For a candidate query $q$ and the reference $q^\star$,

Floats compare to two decimals, column order is free, and row order counts only when $q^\star$ has ORDER BY. Training is GRPO [2] on a LoRA adapter [5] with TRL [7] and vLLM generation [6]. For a prompt with $G$ samples and rewards $r_1, \dots, r_G$, the advantage of sample $j$ is the group-normalized reward $A_j = (r_j - \operatorname{mean}(r)) / \operatorname{std}(r)$, and the loss carries a KL penalty against the base model with coefficient $\beta$, listed per round below.

• r1, from base: GRPO 100 steps, lr 2e-5, beta 0.04, HF generate.
• r2, from r1: GRPO 200 steps, lr 5e-5, beta 0.01.
• sft-think, from base: self-distillation on 199 verified traces, hosted SFT, 2 epochs.
• r3, from r2: GRPO 1,000 steps, lr 2e-5, beta 0.01, vLLM generation, 8 prompts per generate.
• r4, from r3: GRPO 1,000 more steps, same settings.

Correction on record: before SDK 0.51, simulate(tasks=...) drafted a tool surface and sent it to the policy, so 42 of r3's 560 first-pass replies were tool calls scored as failures. The r3 row is the clean re-measure, and the polluted files are under raw/with-drafted-tools/.

The method follows the RLHF book [1]: verifiable rewards (§7), the over-optimization symptoms we watched for in the Qwen rounds (§14), and the evaluation protocol of a fixed held-out split with an interval on the paired difference (§16).

Code: recipes/04-train/text-to-sql. Data: zero-proof-ai/text-to-sql-shop, configs eval-base to eval-r4, eval-nemotron-8b-base, eval-nemotron-8b-r1. Adapters zero-proof-ai/text-to-sql-shop-r1 to -r4 and -nemotron-8b-r1.

References

1. Lambert, N. (2025). Reinforcement Learning from Human Feedback. arXiv:2504.12501. Online at rlhfbook.com.
2. Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y. K., Wu, Y. and Guo, D. (2024). DeepSeekMath: Pushing the limits of mathematical reasoning in open language models. arXiv:2402.03300.
3. Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. de O., Kaplan, J., et al. (2021). Evaluating large language models trained on code. arXiv:2107.03374.
4. Efron, B. (1979). Bootstrap methods: Another look at the jackknife. The Annals of Statistics, 7(1), 1-26.
5. Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L. and Chen, W. (2022). LoRA: Low-rank adaptation of large language models. International Conference on Learning Representations. arXiv:2106.09685.
6. Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H. and Stoica, I. (2023). Efficient memory management for large language model serving with PagedAttention. Proceedings of the 29th ACM Symposium on Operating Systems Principles. arXiv:2309.06180.
7. von Werra, L., Belkada, Y., Tunstall, L., Beeching, E., Thrush, T., Lambert, N., Huang, S., Rasul, K. and Gallouédec, Q. (2020). TRL: Transformer Reinforcement Learning [software]. github.com/huggingface/trl.
8. whilehq (2026). whileai SDK [software]. Apache 2.0. github.com/whilehq/whileai-sdk.

Run it

pip install "whileai>=0.51" "psycopg[binary]" openai anthropic
git clone https://github.com/whilehq/whileai-sdk
cd whileai-sdk/recipes/04-train/text-to-sql
python rollout.py --model qwen3-4b --split holdout --k 4
python build.py

The README has the database setup, the Modal command for training, and delta.py for the paired score.