Distill without a reward

The short version. Reinforcement learning needs a reward. Distillation does not: a teacher scores every token the student writes, and the student moves toward it. We ran both on one task, from one starting model, for the same twenty steps. A frozen stronger model as the teacher reached the same held-out score as the reward. The same model shown the answer as its own teacher moved a fifth as far, because a 0.6B model barely uses a hint. Both calls are two lines in the SDK, and the config writer says which knobs the trainer reads.

The three teachers

The student is a public 0.6B checkpoint that has learned to reverse a sentence character by character, badly. The task is reverse-text, a taskset bundled with prime-rl, scored by how much of the reply matches the true reversal. We hold out the last 128 prompts and never train on them.

• GRPO reads the score. A reply is pushed up when it beats the other replies to the same prompt [1].
• OPD reads a teacher. A frozen, stronger checkpoint of the same family sits on the first GPU and reports how likely it found each token the student wrote. The student moves toward tokens the teacher preferred [2, 3].
• OPSD reads itself. The teacher is the student with the answer pasted into its prompt, which the student never sees. Same per-token signal, no second model [4, 5, 6].

import whileai as wai
 
teacher = wai.Endpoint(
    url="http://localhost:8001/v1",
    model="PrimeIntellect/Qwen3-0.6B-Reverse-Text-RL",
)
 
opd = wai.OPD(teacher)  # reverse KL per token, top_k 32, 4 samples a prompt
opsd = wai.OPSD(
    privileged="answer"
)  # the task field the teacher alone sees, 1 sample a prompt
cfg = wai.prime_rl_config(
    "reverse-text",
    opd,
    model="PrimeIntellect/Qwen3-0.6B-Reverse-Text-SFT",
    out="opd.toml",
)
print(cfg)  # reads / ignores / warnings / uv run rl @ opd.toml

Every default is a named constant with its paper. A chat API as the teacher is refused with the reason (the teacher is scored on the student's tokens, which needs a server that returns log-probabilities). A knob prime-rl does not honor is listed under ignores, not dropped.

What happened

The number that explains OPSD is the one the distillation arms minimize: the logged per-token gap between teacher and student. For OPD it started at -0.29 and closed to -0.11 over the run. For OPSD it sat near -0.08 from the first step to the last. Showing this model the answer barely changed what it predicted for its own tokens, so there was little to learn from. That is the in-context-learning floor the self-distillation papers report near 7B parameters [4, 5], and the warning the config writer prints before the run.

Head to head at step 20 on the same prompts, OPD against GRPO is inside the noise band. A frozen teacher and no reward reached the number the reward reached. OPD paid for a second served model; GRPO paid for a verifier. Where a program can score the task, the reward is the cheaper signal. Where nothing can, the teacher is the one that exists.

What this teaches about post-training

Distillation is an order of magnitude cheaper than RL on the reasoning benchmarks Qwen3 reports [3], and the signal lives on the teacher's most likely tokens [7, 8]. Self-distillation earns its keep where a reward has no gradient, on prompts every sample gets right or wrong [4, 5], and it costs points on thinking models [9]. Before either, check that the teacher beats the student on the held-out set and that the hint changes the teacher. The library will run both checks for you next; today it warns and the log shows the answer.

For researchers

Exact reversals stayed at zero in every arm; the LCS ratio is the result. One run per arm: the intervals are paired over prompts, not over seeds.

Run it

pip install whileai modal
git clone https://github.com/whilehq/whileai-sdk
cd whileai-sdk/recipes/04-train/prime-rl
python run.py --validate      # dry-run the three configs in the prime-rl image
python run.py                 # spawn the three arms on your Modal account
python run.py --collect       # results.json with the paired deltas

The guide is at docs.withwhile.com/distillation.

References

1. Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y. K., Wu, Y. and Guo, D. (2024). DeepSeekMath: Pushing the limits of mathematical reasoning in open language models. arXiv:2402.03300.
2. Agarwal, R., Vieillard, N., Zhou, Y., Stanczyk, P., Ramos, S., Geist, M. and Bachem, O. (2024). On-policy distillation of language models: Learning from self-generated mistakes. International Conference on Learning Representations. arXiv:2306.13649.
3. Yang, A., et al. (2025). Qwen3 technical report. arXiv:2505.09388.
4. Shenfeld, I., Damani, M., Hübotter, J. and Agrawal, P. (2026). Self-distillation enables continual learning. arXiv:2601.19897.
5. Hübotter, J., et al. (2026). Reinforcement learning via self-distillation. International Conference on Machine Learning. arXiv:2601.20802.
6. Zhao, S., et al. (2026). Self-distilled reasoner: On-policy self-distillation for large language models. arXiv:2601.18734.
7. Li, Y., et al. (2026). Rethinking on-policy distillation of large language models: Phenomenology, mechanism, and recipe. arXiv:2604.13016.
8. Fu, Y., et al. (2026). Revisiting on-policy distillation: Empirical failure modes and simple fixes. arXiv:2603.25562.
9. Kaur, S., Ri, N., He, H., Fowl, L. and Arora, S. (2026). Rethinking on-policy self-distillation for thinking models. arXiv:2607.05184.
10. Prime Intellect Team (2025). INTELLECT-2: A reasoning model trained through globally decentralized reinforcement learning. arXiv:2505.07291. prime-rl: github.com/PrimeIntellect-ai/prime-rl.