Swarm the prompts GRPO throws away

The short version. GRPO learns from the spread inside a group of rollouts, so a prompt where all eight fail teaches it nothing and gets dropped. Those are your hardest prompts, and they vanish from the training set. We took 223 of them and gave each one 24 more samples four ways. Every way rescued about one task in ten, and the differences sit inside the noise of re-running the same arm. The caveat is the model: a 4B model with thinking off barely passes a single sample test on these problems, so there was little for the swarm to share.

Where the training set comes from

A reinforcement learning run on a verifier does not train on prompts. It trains on groups: several rollouts of one prompt, scored pass or fail, with the passing ones pushed up relative to the failing ones [1]. If the whole group passes or the whole group fails, there is nothing to push, so the prompt contributes no gradient and dynamic sampling skips it [2]. The SDK's select(mode="rl") keeps the 20 to 80% band for the same reason.

That rule is right and it has a cost. The prompts that fail every time are the ones the model most needs, and they are exactly the ones the training set never contains. In our base pass, 223 of 275 tasks were in that pile.

A swarm is a different way to spend the same samples

Particle swarm optimization is a population search from 1995 [3]. Each particle keeps its own best position, looks at its neighbours' best, and moves toward a mix of the two. The part that transfers to language models is not the arithmetic. It is the social structure, and the one question the field studied for two decades: who sees whom.

We ran it in prose. A particle is one attempt at the problem. Its position is the program it wrote. Its fitness is how many of the visible tests that program passes. Each round, the model gets its own best program, the first visible test it failed with the expected and actual output, and one more thing that depends on the topology:

• Solo sees nothing else. This is the swarm with the social term off.
• Ring sees the best program of its two neighbours [4].
• Star sees the best program in the whole swarm.

Eight particles, three rounds, 24 samples. Resampling gets 24 fresh independent tries. Every arm stops on a task the moment one program passes the hidden tests, which no arm ever sees.

What happened

Feedback alone did nothing. Solo refinement matched resampling. Sharing added six rescues for either topology, and the two resample re-runs moved by more than that between seeds. On the tasks nobody rescued, the best program passes 8% of the visible tests on average. The fitness the swarm moves on was almost always zero, and a particle shown a neighbour's zero-fitness program has no direction to move in.

The training set did move. Across the six runs, 43 of the 223 dropped prompts got at least one program that passes every hidden test. The four arms alone wrote 112 distinct passing programs on 37 tasks into a file of bare rows: the problem in, the program out, no swarm context.

What this teaches about post-training

The training set is not fixed. Which prompts carry signal depends on how much you sample, and on this model it did not depend on how you condition the later samples. QuestA rescues the same prompts with partial reference solutions as hints [5]; the swarm tried to write its own hints and, at this fitness floor, could not. Model Swarms ran PSO over adapter weights [6], where fitness is a held-out score and always graded. A swarm needs a graded fitness. Pass or fail on a task the model cannot touch is not one.

The rescued rows carry a warning label. They were written with a neighbour's program in the prompt, so they are off-policy for the bare prompt and cannot go through the GRPO importance ratio as plain rollouts. They enter as privileged data: rejection-sampled SFT, or the on-policy distillation we compared last week. That training run is the next post, and it does not need the swarm to win first.

For researchers

Run it

pip install whileai pyarrow
git clone https://github.com/whilehq/whileai-sdk
cd whileai-sdk/recipes/01-simulate/swarm-rescue
python run.py --dry-run          # three toy tasks, a fake model, no key
python run.py --limit 20         # twenty tasks on the hosted model
python run.py --noise-runs 2     # every task, four arms, the noise band

References

1. Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y. K., Wu, Y. and Guo, D. (2024). DeepSeekMath: Pushing the limits of mathematical reasoning in open language models. arXiv:2402.03300.
2. Yu, Q., et al. (2025). DAPO: An open-source LLM reinforcement learning system at scale. arXiv:2503.14476.
3. Kennedy, J. and Eberhart, R. (1995). Particle swarm optimization. Proceedings of IEEE International Conference on Neural Networks, 1942 to 1948.
4. Kennedy, J. and Mendes, R. (2002). Population structure and particle swarm performance. Proceedings of the IEEE Congress on Evolutionary Computation, 1671 to 1676.
5. Li, J., et al. (2025). QuestA: Expanding reasoning capabilities of LLMs via question augmentation. arXiv:2507.13266.
6. Feng, S., Wang, Z., Wang, Y., Ebrahimi, S., Palangi, H., Miculicich, L., Kulshrestha, A., Rauschmayr, N., Choi, Y., Tsvetkov, Y., Lee, C.-Y. and Pfister, T. (2024). Model Swarms: Collaborative search to adapt LLM experts via swarm intelligence. arXiv:2410.11163.
7. Li, Y., et al. (2022). Competition-level code generation with AlphaCode. Science 378(6624), 1092 to 1097. arXiv:2203.07814.