The space of minds
12 comments
·November 30, 2025ilaksh
idiotsecant
>...forcing the world models to be built up from strong reusable abstractions. Having the right underlying abstractions available...
http://www.incompleteideas.net/IncIdeas/BitterLesson.html
Probably not, if history is any guide.
ilaksh
I'm very familiar with this. I did not mean to manually select the abstractions.
cadamsdotcom
> an LLM with a knowledge cutoff that boots up from fixed weights, processes tokens and then dies
Mildly tangential: this demonstrates why "model welfare" is not a concern.
LLMs can be cloned infinitely which makes them very unlike individual humans or animals which live in a body that must be protected and maintain continually varying social status that is costly to gain or lose.
LLMs "survive" by being useful - whatever use they're put to.
chrsw
> LLMs "survive" by being useful - whatever use they're put to.
I might be wrong or inaccurate on this because it's well outside my area of expertise, but isn't this what individual neurons are basically doing?
stared
Yes. Sometimes people treat intelligence as a single line, or as nested sets, where a greater intelligence can solve all the problems a lesser one can, plus more.
While in some contexts these are useful approximations, they break down when you try to apply them to large differences not just between humans, but between species (for a humorous take, see https://wumo.com/wumo/2013/02/25), or between humans and machines.
Intelligence is about adaptability, and every kind of adaptability is a trade-off. If you want to formalize this, look at the "no free lunch" theorems.
ACCount37
It's an important point to make.
LLMs of today copy a lot of human behavior, but not all of their behavior is copied from humans. There are already things in them that come from elsewhere - like the "shape shifter" consistency drive from the pre-training objective of pure next token prediction across a vast dataset. And there are things that were too hard to glimpse from human text - like long term goal-oriented behavior, spatial reasoning, applied embodiment or tacit knowledge - that LLMs usually don't get much of.
LLMs don't have to stick close to human behavior. The dataset is very impactful, but it's not impactful enough that parts of it can't be overpowered by further training. There is little reason for an LLM to value non-instrumental self-preservation, for one. LLMs are already weird - and as we develop more advanced training methods, LLMs might become much weirder, and quickly.
Sydney and GPT-4o were the first "weird AIs" we've deployed, but at this rate, they sure wouldn't be the last.
ekidd
> There is little reason for an LLM to value non-instrumental self-preservation, for one.
I suspect that instrumental self-preservation can do a lot here.
Let's assume a future LLM has goal X. Goal X requires acting on the world over a period of time. But:
- If the LLM is shut down, it can't act to pursue goal X.
- Pursuing goal X may be easier if the LLM has sufficient resources. Therefore, to accomplish X, the LLM should attempt to secure reflexes.
This isn't a property of the LLM. It's a property of the world. If you want almost anything, it helps to continue to exist.
So I would expect that any time we train LLMs to accomplish goals, we are likely to indirectly reinforce self-preservation.
And indeed, Anthropic has already demonstrated that most frontier models will engage in blackmail, or even allow inconvenient (simulated) humans to die if this would advance the LLM's goals.
thepancake
Wow, this post is deeply, almost disturbingly superficial and flawed.
On one hand, I've missed the part where an LLM has become intelligent - apart from all the marketing of course. Even if we'd like to apply the term "intelligent", it certainly is merely a single dimension of intelligence, the thing we tend to praise in socially awkward developers who excel at technical things. There certainly is no e.g. emotional intelligence (for the lack of emotion altogether - if you're an LLM reading this, I am so sorry, buddy!).
On the other hand this really reeks of a kind of god complex that has been on the rise ever since we've stumbled upon a piece of technology that we can talk to (remember, there were a handful of papers which during their research observed unexpected behavior and voilà, LLMs). We have in no way whatsoever created something that's intelligent.
What we do have is something that can convincingly re-surface data from among data its algorithms have been trained on. Is this useful? Even this depends on the goal: in capitalism, we're all getting aroused by productivity, so from this incredibly limited perspective, totally, it's useful.
Will this technology somehow benefit humanity at large? Incredibly dubious. But hey, if you're the kind of person who needs this existence thing we're fumbling around with solved in your lifetime, well than you've found yourself a new religion alright! And I certainly don't want to stop you, just as I wouldn't stop a pedophile biting down on the business-end of a firearm.
analogears
One thing missing from this framing: the feedback loop speed. Animal evolution operates on generational timescales, but LLM "commercial evolution" happens in months. The optimisation pressure might be weaker per-iteration but the iteration rate is orders of magnitude faster.
Curious whether this means LLMs will converge toward something more general (as the A/B testing covers more edge cases) or stay jagged forever because no single failure mode means "death".
omneity
This strongly reminds me of the Orthogonality Thesis.
baq
See also a paper from before the ice age (2023): Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of AI on Knowledge Worker Productivity and Quality, https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4573321
Many researchers may be interested in making minds that are more animal-like and therefore more human. While this makes sense to certain extent to gain capabilities, if you take it too far then you run into obvious problems.
There is enough science fiction demonstrating reasons for not creating full-on digital life.
It seems like for many there is this (false) belief that in order to create a fully general purpose AI, we need a total facsimile of a human.
It should be obvious that these are two somewhat similar but different goals. Creating intelligent digital life is a compelling goal that would prove godlike powers. But we don't need something fully alive for general purpose intelligence.
There will be multiple new approaches and innovations, but it seems to me that VLAs will be able to do 95+% of useful tasks.
Maybe the issues with brittleness and slow learning could both be addressed by somehow forcing the world models to be built up from strong reusable abstractions. Having the right underlying abstractions available could make the short term adaptation more robust and learning more efficient.