R-Zero: Self-Evolving Reasoning LLM from Zero Data
9 comments
·September 10, 2025jasonjmcghee
magicalhippo
For those not in the know, that's Generative Adversarial Networks[1], where two neural networks are trained in a competitive way.
One network typically generates tasks for the other, and is rewarded if it manages to make the other network fail the task. The other network is rewarded if it successfully completes the task.
Thus the adversarial network tries to find weaknesses to exploit, and the combined training makes the solving network much stronger. Or at least that's the idea.
[1]: https://en.wikipedia.org/wiki/Generative_adversarial_network
thom
For values of zero quite far above zero.
falcor84
What am I missing? From my skimming, there's zero external data beyond what is needed for the Challenger to generate questions.
thom
An existing trained LLM is an enormous amount of 'data' however it might be encoded. AlphaZero didn't start with Stockfish or a database of games.
magicalhippo
As I understand it the point of the article isn't to train a LLM from scratch, it's to teach a non-reasoning model to reason without additional explicit training data.
tucnak
AlphaZero is oftentimes dragged out to ridicule the so-called "self-play LLM training" techniques, although I don't think these arguments are terribly convincing. You can think of AlphaZero games as effectively synthetic data in adversarial setting; yes, it's easy to produce and verify as the rules of chess are verifiable, so it doesn't require much data on paper. This is not the case for most texts, with some notable exceptions in verifiable domains, where self-play is coincidentally applied most successfully. Thus, you could make an argument that the pre-existing "trained LLM" is merely functioning as a verifier proxy, analogous to the well-defined chess verifier in AlphaZero.
cyberge99
What could go wrong?
magicalhippo
Just don't hook it into the nuclear missile controls. We've seen[1] how that goes[2].
[1]: https://en.wikipedia.org/wiki/Colossus:_The_Forbin_Project
Conceptually, it's effectively a GAN