"The Illusion of Thinking" – Thoughts on This Important Paper

So far they've shown that it fails early if the answer would exceed context size anyways. At least that was the case for the Hanoi problem from what I heard. They somehow failed to notice that though.

You underestimate your abilities, and overestimate the LLM's.

Which version of `bc` was the LLM trained on? Perhaps this is not so important for a stable tool that doesn't change much, but it's critical for many programs and programming libraries. I lost count of the number of times when an LLM generated code that doesn't work for the current version of the library I'm using. In some cases you can tell it to use a specific version, or even feed it the documentation for that version, but that often fails to correct the issue.

And this is without considering the elephant in the room: hallucination. LLMs will often mix up or invent APIs that don't exist. Apologists will say that agents are the solution, and that feeding the error back to the LLM will fix it, but that is often not the case either. That might even make things worse by breaking something else, especially in large-context sessions.

The early OpenAI Instruct models were more like that. The original GPT-3 was only trained to predict the next token, then they used RLHF to make them interpret everything as queries, so that "Explain the theory of gravity to a 6 year old." wouldn't complete to "Explain the theory of relativity to a 6 year old in a few sentences." ChatGPT was probably that, expanded to multi-turn conversation. You can see the beginnings of that ChatGPT style in those examples.

https://openai.com/index/instruction-following/

If you don’t aim for the moon, you’re gonna surely miss it. The number of orbital calculations you need to get right to land on the moon is bonkers /s.

It is not the models anecdote but that of the parent.

And knowing is something even dumb programs do with data. It doesn't imply human cognition

> yeah man, the antrophomorphization is bad.

Unfortunately, it's also the least wrong approach. People who refuse to entertain thinking about LLMs as quasi-humans, are the ones perpetually confused about what LLMs can or cannot do and why. They're always up in arms about prompt injections and hallucinations, and keep arguing those are bugs that need to be fixed, unable to recognize them as fundamental to the model's generality and handling natural language. They keep harping about "stochastic parrots" and Naur's "program theory", claiming LLMs are unable to reason or model in the abstract, despite many published research that lobotomizes the models live and nail concepts as they form and activate.

If you squint and imagine LLM to be a person, suddenly all of these things become apparent.

So I can't really blame people - especially non-experts - for sticking to an approach that's actually yielding much better intuition than the alternatives.

This is a bad faith representation of what people are actually saying which is:

1. As far as we know there is nothing non-physical in the brain (a soul or magical quantum microtubules or whatever). Everything that happens in your head obeys the laws of physics.

2. We definitely are conscious.

3. Therefore consciousness can arise from purely physical processes.

4. Any physical process can be computed. And biological thought appears as far as we can see to simply be an extraordinarily complex computation.

5. Therefore it is possible for a sufficiently fast & appropriately programmed computer to be conscious.

6. LLMs bear many resemblances to human thinking (even though they're obviously different in many ways too), and we've established that computers can be conscious, so you can't trivially rule out LLMs being conscious.

At least for now, I don't think anyone sane says today's LLMs are actually conscious. (Though consciousness is clearly a continuum... so maybe they're as conscious as a worm or whatever.)

The main point is that people trivially dismissing AI consciousness or saying "it can't think" because it's only matrix multiplications are definitely wrong.

It might not be conscious because consciousness requires more complexity, or maybe it requires some algorithmic differences (e.g. on-line learning). But it definitely isn't not-conscious just because it is maths or running on a computer or deterministic or ...

> Right here on HN, where you'd think that most people would be able to think better, there's no shortage of AI fanboys who seriously try to frame the idea that LLMs may be conscious

Why does this surprise you? This is a community hosted by a Silicon Valley venture capital firm, and many of its members are either employed by or own companies in the AI industry. They are personally invested in pushing the narrative being criticized here, so the counter arguments are expected.

Have you used them to code? I think that is the tipping point where the level of interaction is so high, when the models are troubleshooting. That is when you can start to get a little philosophical.

The models make many mistakes and are not great software architects, but watch one or even two models work together to solve a bug and you'll quickly rethink "text transformers".

I think it’s the “just” that they are taking issue with. We are “just” neurons. But we demonstrate interesting emergent behaviors that, in principle, can be reduced to firing neurons but in practice we don’t understand and shouldn’t diminish with the word “just”.

I think the point is we wouldnt say "humans dont reason. It is just chemical and eletrical signals"

It doesn't take conciousness to predict the next word repeatedly.

[dead]

Personal experience?

I don't care about benchmarks, nor LLMs' capability to solve puzzles. This is the first thing AI companies optimize their training for, which is misleading and practically false advertising.

I care about how good LLMs are for helping me with specific tasks. Do they generate code that on the surface appears correct, yet on closer inspection has security and performance issues, is unnecessarily complex, often doesn't even compile, which takes me more time to troubleshoot and fix than if I were to write it myself? Do they explain concepts confidently while being wrong, which I have no way of knowing unless I'm a domain expert? Do they repeat all these issues even after careful re-prompting and with all the contextual information they would need? Does all this waste my time more than it helps?

The answer is yes to all of the above.

So while we can argue whether LLMs can think and reason, my personal experience tells me that they absolutely cannot, and that any similarity to what humans can do is nothing but an illusion.

Worth looking at how many papers the authors have published before assuming they challenge Apple.