Reasoning is not model improvement

There's also the pretty simple observation that the ability to chain tool calls this way is itself a profound model improvement that occurred, in public mainstream foundation models, only a year ago.

OP here. Thanks for the thoughtful reply. Curious if you’ve measured o1’s accuracy and token cost with tool use enabled vs disabled? Wondering if python sandbox gives higher accuracy and lower cost, since internal reasoning chains are longer and pricier.

[Edit] Its probably premature to argue without the above data, but if we assume tool use gives ~100% accuracy and reasoning-only ~90%, then that 10% gap might represent the loss in the probabilistic model: either from functional ambiguity in the model itself or symbolic ambiguity from tokenization?

> A lot of people have been ditching Claude for GPT-5 for coding stuff, and Anthropic held the throne for "best coding model" for well over a year prior to that.

Even if GPT-5 was less capable of a coder than Claude, I'd still not use Claude because of its ridiculous quotas, context window restrictions, slowness, and Anthropic's pedantic stance on AI safety.

Glad to see the pentagonal multiplication prism is just as weird as the addition helix https://arxiv.org/abs/2502.00873

This is what I deem to be more comparable to human reasoning, although in this case it happens at an extremely slow timescale. Ideally real reasoning would have an impact on the weights, although that would be practically impossible(very impractical with the current model architectures) and especially if the conversation is had with the broader public.

That's interesting, though I wonder if what's "baked into the weights" is closer to intuition than reasoning. Once reasoning traces are distilled into weights, the model stops thinking through problems and starts pattern-matching answers. That feels more like a stochastic parrot with intuition than an analytical reasoner.

Exactly. Both the theater script and code are metadata that manipulates entities: characters in a play or variables in memory. There's definitely abstract-level understanding emerging: that's why models can be trained on python, but write code in java. That could be instructions like pseudo-code or the hidden document/theater script you mentioned. That capability jump from GPT3 to o1 is real. But my point is: pure metadata manipulation has hit a ceiling or is moving at a crawling pace since o1. The breakthrough applications (like agentic AI) still depend on the underlying model's ability to generate accurate code. When that capability plateaus, all the clever orchestration on top of it plateaus too.

Wrong on every count, basically.

1. You can enable or disable tool use in most APIs. Generally, tools such as web search and Python interpreter give models an edge. The same is true for humans, so, no surprise. At the frontier, model performance keeps climbing - both with tool use enabled and with it disabled.

2. Model capabilities keep improving. Frontier models of today are both more capable at their peak, and pack more punch for their weight, figuratively and literally. Capability per trained model weight and capability per unit of inference compute are both rising. This is reflected directly in model pricing - "GPT-4 level of performance" is getting cheaper over time.

3. We're 3 years into the AI revolution. If I had ten bucks for every "breakthrough new architecture idea" I've seen in a meanwhile, I'd be able to buy a full GB200 NVL72 with that.

As a rule: those "breakthroughs" aren't that. At best, they offer some incremental or area-specific improvements that could find their way into frontier models eventually. Think +4% performance across the board, or +30% to usable context length for the same amount of inference memory/compute, or a full generational leap but only in challenging image understanding tasks. There are some promising hybrid approaches, but none that do away with "autoregressive transformer with attention" altogether. So if you want a shiny new architecture to appear out of nowhere and bail you out of transformer woes? Prepare to be disappointed.

Reasoning model doesn't imply tool calling – those shouldn't be conflated.

Reasoning just means more implicit chain-of-thought. It can be emulated by non reasoning model by explicitly constructing prompt to perform longer step by step thought process. With reasoning models it just happens implicitly, some models allow for control over reasoning effort with special tokens. Those models are simply fine tuned to do it themselves without explicit dialogue from the user.

Tool calling happens primarily on the client side. Research/web access mode etc made available by some providers (based on tool calling that they handle themselves) is not a property of a model, can be enabled on any model.

Nothing plateaued from where I'm standing – new models are being trained, releases happen frequently with impressive integration speed. New models outperform previous ones. Models gain multi modality etc.

Regarding alternative architectures – there are new ones proposed all the time. It's not easy to verify all of them at scale. Some ideas that are extending current state of art architectures end up in frontier models - but it takes time to train so lag does exist. There are also a lot of improvements that are hidden from public by commercial companies.

1 isn't true. o1 doesn't have access to a Python interpreter unless you explicitly grant it access.

If you call the OpenAI API for o1 and ask it to multiply two large numbers it cannot use Python to help it.

Try this:

    curl https://api.openai.com/v1/responses \
      -H "Content-Type: application/json" \
      -H "Authorization: Bearer $OPENAI_API_KEY" \
      -d '{
        "model": "o1",
        "input": "Multiply 87654321 × 98765432",
        "reasoning": {
          "effort": "medium",
          "summary": "detailed"
        }
      }'

o1 correctly answered the multiplication by running a long multiplication process entirely through reasoning tokens.

>I claim that when o1 multiplies large numbers, it generates Python code rather than calculating internally. I don't have full transparency into o1's internals. Is this accurate?

Both reasoning and non-reasoning models may choose to use the Python interpreter to solve math problems. This isn't hidden from the user; it will show the interpreter ("Analyzing...") and you can click on it to see the code it ran.

It can also solve math problems by working through them step-by-step. In this case it will do long multiplication using the pencil-and-paper method, and it will show its work.

Point 2 is 1000% not true, the models have both gotten better at the overall act of coding, but have also gotten WAY better at USING tools. This isn't tool orchestration frameworks, this is knowing how and when to use tools effectively and it is largely inside the model. I would also say this is a fundamental model capability.

This improved think->act->sense loop that they now form, exponentially increases the possible utility of the models. We are just starting to see this with gpt-5 and the 4+ series of Claude models.

I don't think 2 is true: when OpenAI model won a gold medal in the math olympiads, it did so without tools or web search, just pure inference. Such a feat definitely would not have happened with o1.

I don't really know what you mean by "preserve semantic meaning at the word level." The significant misunderstanding about tokenization present elsewhere in the article is concerning, given that the proposed path forward is to do with replacing tokenization somehow.

All i can say as someone sorta lay is that math isn't an LLM's strength. Having them defer calculations to calculators or python code seems better than it guessing that 1+1 = 2 because it's past data says 1+1 = 2

Heidegger ready-to-hand is another take on this same idea. Something I took to heart years ago and was a big part of my using and contributing to free software as much as possible. Proprietary software is a form of mind control along these lines of thought and I don't like that one bit.

Claude Cod. That would be a neat addition to their portfolio :)

Right, this is something many people seem to miss. LLMs are at best similar to some module in the brain. A full intelligence is almost certainly going to require a mixture of models and a bunch of connections and architecture to tie it all together.

There seems to be a bit of "if your only tool is a hammer" going on with the desire to have a single model do everything.