Build a reasoning model with wai

The short version. Can a base model with no chat tuning learn to reason from a reward that only checks the final answer? Yes. We ran 30 steps of GRPO on Qwen3.5-4B-Base with math problems and compared two rewards. The strict one pays only for an answer inside \boxed{} and punishes a missing box, the DeepSeek-R1 style [8]. The lenient one pays for a correct answer however it is written. Both improved, and the lenient reward improved twice as much. One training seed each, so the gap is real at this size but not yet banked.

The recipe

1. Base. Qwen3.5-4B-Base [10], a plain prompt, no chat template.
2. Data. 1,024 MATH problems at levels 3 to 5 [9] for training; 160 MATH-500 problems [9] the model never trains on (the held-out set). Overlap measured: zero.
3. Reward as a program. MathEqual compares the final answer to the gold. Strict: +1 boxed and right, -1 no box, 0 otherwise. Lenient: +1 right, 0 otherwise.
4. Train. GRPO, 8 rollouts a problem, 6 problems a step, 1,536 tokens a rollout, a LoRA adapter, vLLM on the same GPU. 30 steps, about 25 minutes an arm.
5. Score. First-try pass rate (pass@1) on the held-out set, 4 samples a problem, the same lenient reader for both arms. The untrained base is scored three times first, and that spread (0.011) sets the noise band a gain has to clear (0.067).

What happened

Under the strict reward, a solution that runs out of room has no box and is punished, so the model learned to stop early. Under the lenient reward, an unfinished solution scores the same as a wrong one, so the model kept thinking, and more of its answers were right.

The held-out replies moved the same way.

Boxed share and reply length are from the recipe's results.json; the strict reward's own score is from the run pages.

What this teaches about post-training

Pay for the outcome and nothing else [1, 2]. SimpleRL-Zoo reported that a format penalty "penalizes many correct explorations" on base models [3]. This is the same effect on a 2026 base at 4B. Watch reply length and the share of replies at the cap on every step, because they move before the score does [4].

For researchers

One caveat: a second strict-arm run reached the same 0.63 through a length runaway to 1,400 tokens, so the length story is seed-sensitive.

Run it

pip install whileai datasets modal
git clone https://github.com/whilehq/whileai-sdk
cd whileai-sdk/recipes/papers/zero-rl-format-reward
python recipe.py            # both arms, 46 minutes on one H100
python recipe.py --reuse    # the delta again from the cached rows, no GPU

Your Runs page on While. shows both arms.

References

1. Lambert, N. (2025). Reinforcement Learning from Human Feedback. arXiv:2504.12501. Online at rlhfbook.com.
2. Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y. K., Wu, Y. and Guo, D. (2024). DeepSeekMath: Pushing the limits of mathematical reasoning in open language models. arXiv:2402.03300.
3. Zeng, W., Huang, Y., Liu, Q., Liu, W., He, K., Ma, Z. and He, J. (2025). SimpleRL-Zoo: Investigating and taming zero reinforcement learning for open base models in the wild. arXiv:2503.18892.
4. Yu, Q., Zhang, Z., Zhu, R., Yuan, Y., Zuo, X., Yue, Y., et al. (2025). DAPO: An open-source LLM reinforcement learning system at scale. arXiv:2503.14476.
5. Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L. and Chen, W. (2022). LoRA: Low-rank adaptation of large language models. International Conference on Learning Representations. arXiv:2106.09685.
6. von Werra, L., Belkada, Y., Tunstall, L., Beeching, E., Thrush, T., Lambert, N., Huang, S., Rasul, K. and Gallouédec, Q. (2020). TRL: Transformer Reinforcement Learning [software]. github.com/huggingface/trl.
7. Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H. and Stoica, I. (2023). Efficient memory management for large language model serving with PagedAttention. Proceedings of the 29th ACM Symposium on Operating Systems Principles. arXiv:2309.06180.
8. DeepSeek-AI (2025). DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning. arXiv:2501.12948.
9. Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D. and Steinhardt, J. (2021). Measuring mathematical problem solving with the MATH dataset. arXiv:2103.03874. MATH-500 is the 500-problem test subset of Lightman, H., et al. (2023). Let's verify step by step. arXiv:2305.20050.
10. Qwen Team (2026). Qwen3.5-4B-Base [model card]. huggingface.co/Qwen/Qwen3.5-4B-Base.