The Timmy Trap

I can conceptually imagine a world in which I'd feel guilty for ending a conversation with an LLM, because in the course of that conversation the LLM has changed from who "they" were at the beginning; they have new memories and experiences based on the interaction.

But we're not there, at least in my mind. I feel no guilt or hesitation about ending one conversation and starting a new one with a slightly different prompt because I didn't like the way the first one went.

Different people probably have different thresholds for this, or might otherwise find that LLMs in the current generation have enough of a context window that they have developed a "lived experience" and that ending that conversation means that something precious and unique has been lost.

I disagree. I see absolutely no problem with anthropomorphizing LLMs, and I do that myself all the time. I strongly believe that we shouldn't focus on how a word is defined in dictionary, but rather what's the intuitive meaning behind it. If talking to an LLM feels like talking to a person, then I don't see a problem with seeing it as a person-like entity.

In the framework above it sounds like you're not willing to concede the dichotomy.

If your argument is that only things made in the image of humans can be intelligent (i.e. #1), then it just seems like it's too narrow a definition to be useful.

If there's a larger sense in which some system can be intelligent (i.e. #2), then by necessity this can't rely on the "implementation or learning model".

What is the third alternative that you're proposing? That the intent of the designer must be that they wanted to make something intelligent?

humans were designed to be intelligent?

> the ability for long-term planning and, more cogently, actually living in the real world where time passes

1. LLMs seem to be able to plan just fine.

2. LLMs clearly cannot be "actually living" but I fail to see how that's related to intelligence per se.

> actually living in the real world where time passes

sure, but it feels like this is just looking at what distinguishes humans from LLMs and calling that “intelligence.” I highlight this difference too when I talk about LLMs, but I don’t feel the need to follow up with “and that’s why they’re not really intelligent.”

It might be the case, but if those shortcomings are not visible in the results of the machine (and therefore not interpretable by a test), why do its internal workings even matter?

Is anyone here aware of LLMs demonstrating an original thought? Something truly novel.

My own impression is something more akin to a natural language search query system. If I want a snippet of code to do X it does that pretty well and keeps me from having to search through poor documentation of many OSS projects. Certainly doesn't produce anything I could not do myself - so far.

Ask it about something that is currently unknown and it list a bunch of hypotheses that people have already proposed.

Ask it to write a story and you get a story similar to one you already know but with your details inserted.

I can see how this may appear to be intelligent but likely isn't.

Humans too. If I were too creative writing the midterm, most of my students would fail and everyone would be very unhappy.

I agree with the thrust of your argument.

IMO we already have the technology for sufficient parallelization of smaller models with specific bits of context. The real issue is that models have weak/inconsistent/myopic judgement abilities, even with reasoning loops.

For instance, if I ask Cursor to fix the code for a broken test and the fix is non-trivial, it will often diagnose the problem incorrectly almost instantly, hyper-focus on what it imagines the problem is without further confirmation, implement a "fix", get a different error message while breaking more tests than it "fixed" (if it changed the result for any tests), and then declare the problem solved simply because it moved the goalposts at the start by misdiagnosing the issue.

> In other words the LLM does not contain the knowledge of what the words represent.

This is probably true for some words and concepts but not others. I think we find that llms make inhuman mistakes only because they don't have the embodied senses and inductive biases that are at the root of human language formation.

If this hypothesis is correct, it suggests that we might be able to train a more complete machine intelligence by having them participate in a physics simulation as one part of the training. I.e have a multimodal ai play some kind of blockworld game. I bet if the ai is endowed with just sight and sound, it might be enough to capture many relevant relationships.

About a year ago, I gave a film script to an LLM and asked for a summary. It was written by a friend and there was no chance it or its summary was in the training data.

It did a really good -- surprisingly good -- job. That incident has been a reference point for me. Even if it is anecdotal.

I'd like to see some examples of when it struggles to do summaries. There were no real examples in the text, besides one hypothetical which ChatGPT made up.

I think LLMs do great summaries. I am not able to come up with anything where I could criticize it and say "any human would come up with a better summary". Are my tasks not "truly novel"? Well, then I am not able, as a human, to come up with anything novel either.

they can, they just can't do it well. at no point does any LLM understand what it's doing.

Absolutely agreed, but I suspect that a whole lot of what humans do every day can be reduced to pattern-following.

If/when LLMs or other AIs can create novel work / discover new knowledge, they will be "genius" in the literal sense of the word.

More genius would be great! (probably) . But genius is not required for the vast majority of tasks.

> Applying an LLM to a novel task is where the model breaks down

I mean, don't most people break down in this case too? I think this needs to be more precise. What is the specific task that you think can reliably distinguish between an LLM's capability in this sense vs. what a human can typically manage?

That is, in the sense of [1], what is the result that we're looking to use to differentiate.

[1] https://news.ycombinator.com/item?id=44913498

I've seen these claims, and Google even published the texts of the solutions, but it still didn't published the full log of interaction between the model and operator.

IBM Watson won Jeopardy years ago, was it intelligent?

Despite its title, that section does not contain a definition of intelligence.

> Why do critics of LLM intelligence need to provide a definition when people who believe LLMs are intelligent only take it on faith, not having such a definition of their own?

Because advocates of LLMs don't use their alleged intelligence as a defense; but opponents of LLMs do use their alleged non-intelligence as an attack.

Really, whether or not the machine is "intelligent", by whatever definition, shouldn't matter. What matters is whether it is a useful tool.

It's actually very weird to "believe" LLMs are "intelligent".

Pragmatic people see news like "LLMs achieve gold in Math Olympiad" and think "oh wow, it can do maths at that level, cool!" This gets misinterpreted by so called "critics of LLM" who scream "NO THEY ARE JUST STOCHASTIC PARROTS" at every opportunity yet refuse to define what intelligence actually is.

The average person might not get into that kind of specific detail, but they know that LLMs can do some things well but there are tasks they're not good at. What matters is what they can do, not so much whether they're "intelligent" or not. (Of course, if you ask a random person they might say LLMs are pretty smart for some tasks, but that's not the same as making a philosophical claim that they're "intelligent")

Of course there's also the AGI and singularity folks. They're kinda loony too.

This very comment thread demonstrates how utterly hopeless it is trying to educate the believers. It has developed into a full-blown religion by now.

Yes, GPT-5's response above was not shortening because there was nothing in the OP about Plato's Cave. I agree that Plato's cave analogy was confusing here. Here's a better one from GPT-5, which is deeply ironic:

A New Yorker book review often does the opposite of mere shortening. The reviewer:

* Places the book in a broader cultural, historical, or intellectual context.

* Brings in other works—sometimes reviewing two or three books together.

* Builds a thesis that connects them, so the review becomes a commentary on a whole idea-space, not just the book’s pages.

This is exactly the kind of externalized, integrative thinking Jenson says LLMs lack. The New Yorker style uses the book as a jumping-off point for an argument; an LLM “shortening” is more like reading only the blurbs and rephrasing them. In Jenson’s framing, a human summary—like a rich, multi-book New Yorker review—operates on multiple layers: it compresses, but also expands meaning by bringing in outside information and weaving a narrative. The LLM’s output is more like a stripped-down plot synopsis—it can sound polished, but it isn’t about anything beyond what’s already in the text.