The LLMentalist Effect

I think it's fair to argue that part of human intelligence is actually just a statistical matching. The best example which comes to mind is actually grammar. Grammar has a very complex set of rules, however most people cannot really name them or describe them accurately; instead, they just know whether or not a sentence _sounds_ correct. This feels a lot like the same statistical matching performed by LLMs. An individual's reasoning iterates in their mind what words follow each other, and what phrases are likely.

Outside of grammar, you can hear a lot of this when people talk; their sentences wander, and they don't always seem to know ahead of time where their sentences will end up. They start anchored to a thought, and seem to hope that the correct words end up falling into place.

Now, does all thought work like this? Definitely not, and more importantly, there are many other facets of thought which are not present in LLMs. When someone has wandered badly when trying to get a sentence out, they are often also able to introspect and see that they failed to articulate their thought. They can also slow down their speaking, or pause, and plan out ahead of time; in effect, using this same introspection to prevent themselves from speaking poorly in the first place. Of course there's also memory, consciousness, and all sorts of other facets of intelligence.

What I'm on the fence about is whether this point, or your point actually detracts from the author's argument.

Yeah, the basic premise is off because LLM responses are regularly tested against ground truth (like running the code they produce), and LLMs don't get to carefully select what requests they fulfill. To the contrary they fulfill requests even when they are objectively incapable of answering correctly, such as incomplete or impossible questions.

I do think there is a degree of mentalist-like behavior that happens, maybe especially because of the RLHF step, where the LLM is encouraged to respond in ways that seem more truthful or compelling than is justified by its ability. We appreciate the LLM bestowing confidence on us, and rank an answer more highly if it gives us that confidence... not unlike the person who goes to a spiritualist wanting to receive comforting news of a loved one who has passed. It's an important attribute of LLMs to be aware of, but not the complete explanation the author is looking for.

Which aspect of how a human leg works is a truck tire similar to?

And only to your eyes and those you force your vision onto. The rest of the universe never sees it. You don’t exist to much of the universe (if a tree falls and no one is around to hear it, you understand what I mean).

So you simultaneously exist and don’t exist. Sorry about this, your post took me on this tangent.

I don't disagree that "is it useful" is the important question.

The amount of money being invested is very clearly disproportionate to the current utility, and much of it is obviously based on the premise that LLMs can (or will soon) be able to think.

I think at best, there's a wide gulf between what LLMs can actually do, and what people _believe_ they can do. This is not to say that LLMs are not useful, but just that people broadly and regularly misunderstand their usefulness. This is not necessarily the fault of LLMs, but it does render their usage and adoption a bit problematic.

It has to both be useful and economical. If that answer cost $2 to get you and a search+youtube video would have been just as effective and much cheaper, then it's possible that the new way to get the answer isn't significantly better than the old way.

The graveyards of startups are littered with economically infeasible solutions to problems.

So far it seems LLM-based systems are still reaching for a use.

> Decades of research were unable to answer this.

From the article:

> The field of AI research has a reputation for disregarding the value of other fields [...] It’s likely that, being unaware of much of the research in psychology on cognitive biases or how a psychic’s con works, they stumbled into a mechanism and made chatbots that fooled many of the chatbot makers themselves.

Just because cognitive scientists don't know everything about how intelligence works (or on what it is) doesn't mean that they know nothing. There has been a lot of progress in cognitive science, in the last decade in particular on reasoning.

> based on ill-defined terms like "intelligence".

The whole discussion is about "artificial intelligence". Arguably AI researchers ought to have a fairly well defined stance of what "intelligence" means and can't use a trick like "nobody knows what intelligence is" to escape criticism.

[Back in the 1980s]

You should try to play chess yourself, and then tell me you think these things aren't intelligent.

LLMs don't do too well on those ARC-AGI problems. Even though they're pretty easy for a person.

Why would it have to be one or the other? Yes, it's been proven LLMs do create world models, how good they are is a separate matter. There still could be goal misalignment, especially when it comes to RLHF.

If the model has in its internal world model knowledge it likely does not know how to solve a coding question, but the RLHF stage has reviewers rate refusals lower, it would in turn force its hand when it comes to tricks it knows it can pull based on its model of human reviewers. It can only implement the surface level boilerplate and pass that off as a solution, write its code in APL to obfuscate its lack of understanding, or keep misinterpreting the problem into a simpler one.

A psychic that read on ten thousand biographies might start to recall them, or he might interpolate the blanks with a generous dose of BS, or more likely do both in equal measure.

> Nowadays models can solve original math puzzles much of the time

Isn't that usually by not even trying, and delegating the work regular programs?

how come you elected not to enumerate *any* of the real issues you expected?

Interesting. Resonate with people is what a mentalist does. More generically a good bedside manner is what a good doctor does and it's the same thing. Many times we like being cold read and comforted and comfortable!

Agreed on the experiments. What would they look like? Can a chat bot give the same info without any bedside manner?

Well they have a book to sell, at the bottom of the article.

Absolutely -- there is a whole cottage industry of people who know that LLMs are big news and decide to chime in, without ever considering that maybe this is a scientific discussion, not a cultural/"take-driven" one. Bluesky in particular is probably their biggest source of income!

You can spot them easily, because instead of critiquing some specific thing and sticking to it, they can't resist throwing in "obviously, LLMs are all 100% useless and anyone who says otherwise is a Tech Bro" somewhere. Like:

  completely unknown processes that have no parallel in the biological world.

[flagged]

At this point I think it is because of a type of religious sentimentality about the sanctity of human reasoning.

To take your analogy even further, it is like asking when is the plane going to improve enough that it can really fly by flapping it's wings.

You're right! AI makes us ask really important questions about our own intelligence. I think it will lead to greater recognition that we are first and foremost animals: creatures of intention and action. I think we've put way too much emphasis on our intellectual dimension in the last few hundred years. To the point that some people started to believe that was what we are.

Perhaps we're confusing intelligence with awareness.

What LLMs seem to emulate surprisingly well is something like a person's internal monologue, which is part of but not the whole of our mind.

It's as if it has the ability to talk to itself extremely quickly and while plugged directly into ~all of the written information humanity has ever produced, and what we see is the output of that hidden, verbally-reasoned conversation.

Something like that could be called intelligent, in terms of its ability to manipulate symbols and rearrange information, without having even a flicker of awareness, and entirely lacking the ability to synthesise new knowledge based on an intuitive or systemic understanding of a domain, as opposed to a complete verbal description of said domain.

Or to put it another way - it can be intelligent in terms of its utility, without possessing even an ounce of conscious awareness or understanding.