Emerging reasoning with reinforcement learning

> Backtracking assumes depth-first search, which isn't strictly needed as you could explore all possible options in parallel in a breadth-first manner

You could in theory, but it'd be massively/prohibitively more expensive - exploring a whole tree just to end up using a single branch. It'd be like trying to have a computer play chess by evaluating EVERY branching position out to some fixed depth, rather than using MCTS to prune away unpromising lines.

Not only that, but reasoning in general (of which LLM-based reasoning is only a limited case) isn't just about search - it can also require exploration and acquisition of new knowledge if you run into an impasse that your past experience can't handle. If AI systems hope to achieve human-level AGI, they will need to change to support continuous learning and exploration as a part of reasoning, which naturally means more of a depth-first approach (continue until you hit an impasse) with backtracking.

You can say that hallucination is due to gaps in the training set, but of course humans don't have that problem because we know what we know, and have episodic memories of when/where/how we learned things. LLMs have none of that.

I found the following thread more insightful than my original comment (wish I could edit that one). A research explains why RL didn't work before this: https://x.com/its_dibya/status/1883595705736163727

The group in this article used straight and simple PPO, so I guess GRPO isn't required.

My hypothesis is that everyone was just so stunned by oai's result so most just decided to blindly chase it and do what oai did (i.e. scaling up). And it's only after o1 people started seriously trying other ideas.

I don't have any intuition here and am in no way qualified, but my read of the paper was that GRPO was mainly an optimization to reduce cost & GPUs when training (by skipping the need to keep another copy of the LLM in memory as the value network), but otherwise any RL algorithm should have worked? I mean it seems R1 uses outcome rewards only and GRPO doesn't do anything special to alleviate reward sparsity, so it feels like it shouldn't affect viability too much.

Also on the note of RL optimizers, if anyone here is familiar with this space can they comment on how the recently introduced PRIME [1] compares to PPO directly? Their description is confusing since the "implicit PRM" they introduce which is trained alongside the policy network seems no different from the value network of PPO.

[1] https://github.com/PRIME-RL/PRIME

How do we know that they didn't see it? Their work is much more secret now. Isn't it possible that o1 and o3 rely on something similar maybe with some additions. Same for the gemini thinking models.

My point it that OpenAI and google might have been working with very similar approaches for months.

"Search" a class of fairly well defined algorithms, "reasoning" is vague / ambiguous. If reasoning can be reduced to some kind of search, that makes its meaning more precise and understandable.

It would be startling if humans could reason this way with two kilos of meat consuming 40w. Even more surprising given that we get answers in ~1s with a cross brain comms time of about 0.25s

To me it's clear that human reasoning is different from a massive search of a latent space. I can say that when I am thinking I maybe try half a dozen ideas or scenarios, but very rarely more. I can't say where those ideas come from or how I make them up though. Maybe we can't frame what it is and how it works with human languages though, which might make it seem magical in some way.

Or maybe there's a good framing that I don't know - would love to learn!

Yes, it has at least two distinct parts: consciousness and sub. First is visible to 'us', it's inner monologue or vision, or other senses. But we don't 'know' what happens in subconsciousness. The answer just pops up from nowhere. 'Reasoning' LLMs for now have the first part. The 'sub' part is questionable, depends how you look at latent space.

As you're demonstrating, it depends on the human.

no, humans do the same thing - even the "intuition" of things to try are probably the results of searches, but we don't have conscious access to them. certainly it's the simplest explanation that fits the observations.

It means being good at first order predicate logic. And possibly higher order too when you consider `call/n` and lambdas. It means being good at generalization, at reasoning in causal terms, at understanding structure and grammar, at encoding problems as graphs and querying them for solutions, and much more.

Basically it's what current LLMs lack. They're good at spewing coherent text but they lack the building blocks of reason, which are made of logic, and which confer the quality of being consistent. A implies B.

>Frontier AI systems have surpassed the self-replicating red line

https://arxiv.org/abs/2412.12140

We've had "reasoning" machines for a long time - I learned chess playing against a computer in the 1980's.

But we don't have reasoning that can be applied generally in the open world yet. Or at least I haven't seen it.

In terms of society it should be easy to track if this is true or not. Healthcare and elder care settings will be a very early canary of this because there is huge pressure for improvement and change in these. General reasoning machines will make a very significant, clear and early impact here. I have seen note taking apps for nurses - but not much else so far.

It's not intelligence that separates us from machines, but "connectedness to the whole." A machine becomes alive the moment it's connected to the whole, the moment it becomes driven not by an RNG and rounding errors, but by a spirit. Similarly, a man becomes a machine the moment he loses this connection.

The existential dread around AI is due to the fear that this machine will indeed be connected to a spirit, but to an evil one, and we'll become unwanted guests in a machine civilization. Art and music will be forbidden, for it "can't be reasoned about" in machine terms, the nature will be destroyed for it has to no value to the machines, and so on.

It’s not about being afraid, it’s that the auto-reconfiguration of neurons seems advanced to decompile it at this time, and it surprising that LLM, which are just a probabilistic model of guessing the next word, could be capable of actual thought.

The day it happens, we’ll believe it. There are only 100bn neurons in a brain, after all, and many more than this in modern machines, so it is theoretically possible. Just LLMs seemed too simple for that.

The best example I have seen for true AGI benchmarking is ARC-AGI:

https://arcprize.org/

As the ARC-AGI is more trained to we put more of our own expertise and knowledge into the algorithms. When a computer can get human level results on a new ARC-AGI like benchmark (transfer from other intelligence tasks), then we are very close.

Why do you believe that is how humans reason?

Equating human reasoning to regurgitating a single token at a time requires that you pretend there are not trillions of analog calculations happening in parallel in the human mind. How could there not be a massive search included as part of the reasoning process? LLMs do not perform a massive search, or in any way update their reasoning capability after the model is generated.

What's with AI boosters and not viewing other people as human?

That sword cuts both ways.

Personal attacks really make your argument stronger.

I don't think anyone has said LLMs wouldn't make any progress toward AGI, especially of researchers in the field. But a small piece of progress toward AGI is not the same as AGI.

Reasoning is a human, social and embodied activity. TFA is about machines that output text reminiscent of the results of reasoning, but it's obviously fake since the machine is neither human, social or embodied.

It's an attempt at fixing perceived problems with the query planner in an irreversibly compressed database.

"Star Wars" as the race between the USSR and the US to dominate space—landing on the Moon, the first animal in space, the first human in space, etc. It spanned decades and was a huge drain.

And with what prompt was this prompt written? Its prompts all the way down?

So in response it wrote a heroically tall mountain of bullshit, laced with falsehoods of the sort that even some neanderthal natcon would be unable to dream up, then served it as an abysmally long, overdubbed narration to the next Top Gun movie set in the same future as Idiocracy.

The way you taught chain-of-thought before was with supervised fine tuning (SFT). During training, you have to rate every sentence of reasoning the model writes, many times, to nudge it to reason correctly.

But this approach to teach chain-of-thought doesn’t do that. In this post, they take a small model (7B) that already knows math. Then they give it a relatively small number of problems to solve (8k). They use a simple reinforcement learning loop where the only goal is to get the problem right. They don’t care how the model got the right answer, just that it’s correct.

This is part of the “recipe” that DeepSeek used to create R1.

After many iterations, just like DeepSeek, they found that the model has an “aha” moment. It starts emitting chains-of-thought where it wasn’t before. And then it starts getting the math answers right.

This is the gist of it. I don’t fully understand the recipe involved.

Can you teach small models to “think” just using RL? How small can they be? What tasks does this work for? Is just RL best for this? RL+SFT? Everyone’s trying to figure it out.

Why this behavior emerges is an active area of research. What they did is use reinforcement learning, this blog post replicates those findings. The “recipe” is detailed in the R1 paper.

> We demonstrate that the reasoning patterns of larger models can be distilled into smaller models, resulting in better performance compared to the reasoning patterns discovered through RL on small models

I found this statement from the paper to be at odds with what you cited, but I guess they mean SFT+RL would be better than either just SFT and RL

I should have said 'big deal, for better or for worse.' Regardless of whether one thinks it's a good thing, this was a major discovery that turned out to affect a lot of things.

Eloquently said.

Emergent basically means ‘we didn’t design this capability but it’s there’, it’s always been a thing, not sure why you associate it with AGI so strongly.

Do you think its not emergent because you think this behavior was explicitly coded in or you dont think its emergent because you dont like the implications of thinking it?