AGI is an engineering problem, not a model training problem
226 comments
·August 24, 2025cakealert
MadnessASAP
You didn't mention it, but LLMs and co don't have loops. Whereas a brain, even a simple one is nothing but loops. Brains don't halt, they keep spinning while new inputs come in and output whenever they feel like it. LLMs however do halt, you give them an input, it gets transformed across the layers, then gets output.
While you say reinforcement learning isn't a good answer, I think its the only answer.
cakealert
> While you say reinforcement learning isn't a good answer, I think its the only answer.
Possibly, but even if you have sufficient compute to attempt the bruteforce approach I suspect that such a system simply wouldn't converge.
Animal brains are on the edge of chaos, and chaos in gradient descent means vanishing and exploding gradients. So it comes down to whether or not you can have a "smooth" brain.
Ultimately, cracking the biological learning algorithm would be the golden ticket I think. Even the hyper sample-efficient LLM's don't hold a candle to the bright star of sample-efficiency that is the animal brain.
andy99
If you believe the bitter lesson, all the handwavy "engineering" is better done with more data. Someone likely would have written the same thing as this 8 years ago about what it would take to get current LLM performance.
So I don't buy the engineering angle, I also don't think LLMs will scale up to AGI as imagined by Asimov or any of the usual sci-fi tropes. There is something more fundamental missing, as in missing science, not missing engineering.
hnuser123456
Even more fundamental than science, there is missing philosophy, both in us regarding these systems, and in the systems themselves. An AGI implemented by an LLM needs to, at the minimum, be able to self-learn by updating its weights, self-finetune, otherwise it quickly hits a wall between its baked-in weights and finite context window. What is the optimal "attention" mechanism for choosing what to self-finetune with, and with what strength, to improve general intelligence? Surely it should focus on reliable academics, but which academics are reliable? How can we reliably ensure it studies topics that are "pure knowledge", and who does it choose to be, if we assume there is some theoretical point where it can autonomously outpace all of the world's best human-based research teams?
Uehreka
Nah.
The real philosophical headache is that we still haven’t solved the hard problem of consciousness, and we’re disappointed because we hoped in our hearts (if not out loud) that building AI would give us some shred of insight into the rich and mysterious experience of life we somehow incontrovertibly perceive but can’t explain.
Instead we got a machine that can outwardly present as human, can do tasks we had thought only humans can do, but reveals little to us about the nature of consciousness. And all we can do is keep arguing about the goalposts as this thing irrevocably reshapes our society, because it seems bizarre that we could be bested by something so banal and mechanical.
galangalalgol
I think Metzinger nailed it, we aren't conscious at all. We confuse the map for the territory in thinking the model we build to predict our other models is us. We are a collection of models a few of which create the illusion of consciousness. Someone is going to connect a handful of already existing models in a way that gives an AI the same illusion sooner rather than later. That will be an interesting day.
nikkwong
I found it strange that John Carmack and Ilya Sutskever both left prestigious positions within their companies to pursue AGI as if they had some proprietary insight that the rest of industry hadn't caught on to. To make as bold of a career move that publicly would mean you'd have to have some ultra serious conviction that everyone else was wrong or naive and you were right. That move seemed pompous to me at the time; but I'm an industry outsider so what do I know.
And now, I still don't know; the months go by and as far as I'm aware they're still pursuing these goals but I wonder how much conviction they still have.
jasonwatkinspdx
With Carmack it's consciously a dilliante project.
He's been effectively retired for quite some time. It's clear at some point he no longer found game and graphics engine internals motivation, possibly because the industry took the path he was advocating against back in the day.
For a while he was focused on Armadillo aerospace, and they got some cool stuff accomplished. That was also something of a knowing pet project, and when they couldn't pivot to anything that looked like commercial viability he just put it in hibernation.
Carmack may be confident (ne arrogant) enough to think he does have something unique to offer with AGI, but I don't think he's under any illusions it's anything but another pet project.
therobots927
The simple explanation is that they got high on their own supply. They deluded themselves into thinking an LLM was on the verge of consciousness.
ants_everywhere
> there is missing philosophy
I doubt it. Human intelligence evolved from organisms much less intelligent than LLMs and no philosophy was needed. Just trial and error and competition.
solid_fuel
We are trying to get there without a few hundred million years of trial and error. To do that we need to lower the search space, and to do that we do actually need more guiding philosophy and a better understanding of intelligence.
crystal_revenge
The magical thinking around LLMs is getting bizarre now.
LLMs are not “intelligent” in any meaningful biological sense.
Watch a spider modify its web to adapt to changing conditions and you’ll realize just how far we have to go.
LLMs sometimes echo our own reasoning back at us in a way that sounds intelligent and is often useful, but don’t mistake this for “intelligence”
fuckaj
The physical universe has much higher throughput and lower latency than our computer emulating a digital world.
fragmede
A system that self-updates its weights is so obvious the only question is who will be the first to get there?
soulofmischief
It's not always as useful as you think from the perspective of a business trying to sell an automated service to users who expect reliability. Now you have to worry about waking up in the middle of the night to rewind your model to a last known good state, leading to real data loss as far as users are concerned.
Data and functionality become entwined and basically you have to keep these systems on tight rails so that you can reason about their efficacy and performance, because any surgery on functionality might affect learned data, or worse, even damage a memory.
It's going to take a long time to solve these problems.
null
fuckaj
True. In the same way as making noises down a telephone line is the obvious way to build a million dollar business.
danenania
I’m not sure that self-updating weights is really analogous to “continuous learning” as humans do it. A memory data structure that the model can search efficiently might be a lot closer.
Self-updating weights could be more like epigenetics.
tomrod
Aye. Missing are self correction (world models/action and response observation), coherence over the long term, and self-scaling. The 3rd are what all the SV types are worried about, except maybe Yann LeCun who is worried about the first and second.
Hinton thinks the 3rd is inevitable/already here and humanity is doomed. It's an odd arena.
Bukhmanizer
This isn’t really what the bitter lesson says.
whatever1
The counter argument is that we were working with thermodynamics before knowing the theory. Famously the steam engine came before the first law of thermodynamics. Sometimes engineering is like that. Using something that you don’t understand exactly how it works.
justcallmejm
The missing science to engineer intelligence is composable program synthesis. Aloe (https://aloe.inc) recently released a GAIA score demonstrating how CPS dramatically outperforms other generalist agents (OpenAI's deep research, Manus, and Genspark) on tasks similar to those a knowledge worker would perform.
I'd argue it's because intelligence has been treated as a ML/NN engineering problem that we've had the hyper focus on improving LLMs rather than the approach articulated in the essay.
Intelligence must be built from a first principles theory of what intelligence actually is.
joe_the_user
CPS sounds interesting but your link goes to a teaser trailer and a waiting list. It's kind of hard to expect much from that.
null
energy123
What will it scale up to if not AGI? OpenAI has a synthetic data flywheel. What are the asymptotics of this flywheel assuming no qualitative additional breakthrough?
supermatt
What will shouting louder achieve if not wisdom?
richyg840
It's established science (see Chomsky) that a probablistic model will never achieve something approaching AGI
null
bsenftner
AGI, by definition, in its name Artificial General Intelligence implies / directly states that this type of AI is not some dumb AI that requires training for all its knowledge, a general intelligence merely needs to be taught how to count, the basic rules of logic, and the basic rules of a single human language. From those basics all derivable logical human sciences will be rediscovered by that AGI and our next job is synchronizing with it our names for all the phenomenon that the AGI had to name on its own when that AGI self developed all the logical ramifications of our basics.
What is that? What could merely require light elementary education and then it takes off and self improves to match and surpass us? That would be artificial comprehension, something we've not even scratched. AI and trained algorithms are "universal solvers" given enough data, This AGI would be something different, this is understanding, comprehending. Instantaneous decomposition of observations for assessment of plausibility, and then recombination for assessment of combination plausibility - all continual and instant for assessment of personal safety: all that happens in people continually while awake. Be that monitoring of personal safety be for physical or loss of client during sales negotiation. Our comprehending skills are both physical and abstract. This requires a dynamic assessment, an ongoing comprehension that is validating observations as a foundation floor, so a more forward train of thought, a "conscious mind" can make decisions without conscious thought about lower level issues like situational safety. AGI needs all that dynamic comprehending capability, to satisfy its name of being general.
dragonwriter
> AGI, by definition, in its name Artificial General Intelligence implies / directly states that this type of AI is not some dumb AI that requires training for all its knowledge, a general intelligence merely needs to be taught how to count, the basic rules of logic, and the basic rules of a single human language. From those basics all derivable logical human sciences will be rediscovered by that AGI
That's not how natural general intelligences work, though.
bsenftner
Are you sure? Do you require dozens, to hundreds, to thousands of examples before you understand a concept? I expect no. That is because you have comprehension that can generalize a situation to basic concepts which you apply to other situations without effort. You comprehend. AI cannot do that: get the idea from a few, under a half dozen examples if necessary. Often a human needs 1-3 examples before they can generalize any concept. Not AI.
Davidzheng
I think they're saying people generally don't learn language or mathematics by learning the basic rules and deducing everything else
stickfigure
Any human old enough to talk has already experienced thousands of related examples of most everyday concepts.
For concepts that are not close to human experience, yes humans need a comically large number of examples. Modern physics is a third-year university class.
efitz
Nothing that we consider intelligent works like LLMs.
Brains are continuous - they don’t stop after processing one set of inputs, until a new set of inputs arrives.
Brains continuously feed back on themselves. In essence they never leave training mode although physical changes like myelination optimize the brain for different stages of life.
Brains have been trained by millions of generations of evolution, and we accelerate additional training during early life. LLMs are trained on much larger corpuses of information and then expected to stay static for the rest of their operational life; modulo fine tuning.
Brains continuously manage context; most available input is filtered heavily by specific networks designed for preprocessing.
I think that there is some merit that part of achieving AGI might involve a systems approach, but I think AGI will likely involve an architectural change to how models work.
starchild3001
I think this essay lands on a useful framing, even if you don’t buy its every prescription. If we zoom out, history shows two things happening in parallel: (1) brute-force scaling driving surprising leaps, and (2) system-level engineering figuring out how to harness those leaps reliably. GPUs themselves are a good analogy: Moore’s Law gave us the raw FLOPs, but CUDA, memory hierarchies, and driver stacks are what made them usable at scale.
Right now, LLMs feel like they’re at the same stage as raw FLOPs; impressive, but unwieldy. You can already see the beginnings of "systems thinking" in products like Claude Code, tool-augmented agents, and memory-augmented frameworks. They’re crude, but they point toward a future where orchestration matters as much as parameter count.
I don’t think the "bitter lesson" and the "engineering problem" thesis are mutually exclusive. The bitter lesson tells us that compute + general methods win out over handcrafted rules. The engineering thesis is about how to wrap those general methods in scaffolding that gives them persistence, reliability, and composability. Without that scaffolding, we’ll keep getting flashy demos that break when you push them past a few turns of reasoning.
So maybe the real path forward is not "bigger vs. smarter," but bigger + engineered smarter. Scaling gives you raw capability; engineering decides whether that capability can be used in a way that looks like general intelligence instead of memoryless autocomplete.
Sevii
I don't understand how people feel comfortable writing 'LLMs are done improving, this plateau is it.' when we haven't even gone an entire calendar year without seeing improvements to LLM based AI.
hahn-kev
I wonder if the people saying that would agree that they've been improving.
cellis
If we are truly trying to "replace human at work" as the definition of an AGI, then shouldn't the engineering goal be to componentize the human body? If we could component-by-component replace any organ with synthetic ones ( and this is already possible to some degree e.g. hearing aids, neuralinks, pacemakers, artificial hearts ) then not only could we build compute out in such a way but we could also pull humanity forward and transcend these fallible and imminently mortal structures we inhabit. Now, should we from a moral perspective is a completely different question, one I don't have an answer to.
jfim
Not necessarily. For example, early attempts to make planes tried to imitate birds with flapping wings, but the vast majority of modern planes are fixed wing aircraft.
Imitating humans would be one way to do it, but it doesn't mean it's an ideal or efficient way to do it.
jayd16
From the moment I understood the monolithic design of my flesh, it disgusted me.
glitchc
We don't know if AGI is even possible outside of a biological construct yet. This is key. Can we land on AGI without some clear indication of possibility (aka Chappie style)? Possibly, but the likelihood is low. Quite low. It's essentially groping in the dark.
A good contrast is quantum computing. We know that's possible, even feasible, and now are trying to overcome the engineering hurdles. And people still think that's vaporware.
tshaddox
> We don't know if AGI is even possible outside of a biological construct yet. This is key.
A discovery that AGI is impossible in principle to implement in an electronic computer would require a major fundamental discovery in physics that answers the question “what is the brain doing in order to implement general intelligence?”
AIPedant
It is vacuously true that a Turing machine can implement human intelligence: simply solve the Schrödinger equation for every atom in the human body and local environment. Obviously this is cost-prohibitive and we don’t have even 0.1% of the data required to make the simulation. Maybe we could simulate every single neuron instead, but again it’ll take many decades to gather the data in living human brains, and it would still be extremely expensive computationally since we would need to simulate every protein and mRNA molecule across billions of neurons and glial cells.
So the question is whether human intelligence has higher-level primitives that can be implemented more efficiently - sort of akin to solving differential equations, is there a “symbolic solution” or are we forced to go “numerically” no matter how clever we are?
walleeee
> It is vacuously true that a Turing machine can implement human intelligence
The case of simulating all known physics is stronger so I'll consider that.
But still it tells us nothing, as the Turing machine can't be built. It is a kind of tautology wherein computation is taken to "run" the universe via the formalism of quantum mechanics, which is taken to be a complete description of reality, permitting the assumption that brains do intelligence by way of unknown combinations of known factors.
For what it's worth, I think the last point might be right, but the argument is circular.
Here is a better one. We can/do design narrow boundary intelligence into machines. We can see that we are ourselves assemblies of a huge number of tiny machines which we only partially understand. Therefore it seems plausible that computation might be sufficient for biology. But until we better understand life we'll not know.
Whether we can engineer it or whether it must grow, and on what substrates, are also relevant questions.
If it appears we are forced to "go numerically", as you say, it may just indicate that we don't know how to put the pieces together yet. It might mean that a human zygote and its immediate environment is the only thing that can put the pieces together properly given energetic and material constraints. It might also mean we're missing physics, or maybe even philosophy: fundamental notions of what it means to have/be biological intelligence. Intelligence human or otherwise isn't well defined.
tshaddox
> It is vacuously true that a Turing machine can implement human intelligence: simply solve the Schrödinger equation for every atom in the human body and local environment.
Yes, that is the bluntest, lowest level version of what I mean. To discover that this wouldn’t work in principle would be to discover that quantum mechanics is false.
Which, hey, quantum mechanics probably is false! But discovering the theory which both replaces quantum mechanics and shows that AGI in an electronic computer is physically impossible is definitely a tall order.
Davidzheng
i'd argue LLMs and deep learning are much more on the intelligence from complexity side than the nice symbolic solution side of things. Probably the human neuron, though intrinsically very complex, has nice low loss abstractions to small circuits. But on the higher levels, we don't build artificial neural networks by writing the programs ourselves.
b_e_n_t_o_n
It's not even known if we can observe everything required to replicate consciousness.
missingrib
That is only true if consciousness is physical and the result of some physics going on in the human brain. We have no idea if that's true.
manquer
Not necessarily , for a given definition of AGI you could have mathematical proof that it is incomputable similar to how Gödel incompleteness theorems work .
It need not even be incomputable, it could be NP hard and practically be incomputable, or it could be undecidable I.e. a version of the halting problem.
There are any number of ways our current models of mathematics or computation can in theory could be shown as not capable of expressing AGI without needing a fundamental change in physics
throwaway31131
We would also need a definition of AGI that is provable or disprovable.
We don’t even have a workable definition, never mind a machine.
thfuran
Only if we need to classify things near the boundary. If we make something that’s better at every test that we can devise than any human we can find, I think we can say that no reasonable definition of AGI would exclude it without actually arriving at a definition.
tshaddox
We don’t need such a definition of general intelligence to conclude that biological humans have it, so I’m not sure why we’d such a definition for AGI.
slashdave
That question is not a physics question
aorloff
A question which will be trivial to answer once you properly define what you mean by "brain"
Presumably "brains" do not do many of the things that you will measure AGI by, and your brain is having trouble understanding the idea that "brain" is not well understood by brains.
Does it make it any easier if we simplify the problem to: what is the human doing that makes (him) intelligent ? If you know your historical context, no. This is not a solved problem.
tshaddox
> Does it make it any easier if we simplify the problem to: what is the human doing that makes (him) intelligent ?
Sure, it doesn’t have to be literally just the brain, but my point is you’d need very new physics to answer the question “how does a biological human have general intelligence?”
null
epiccoleman
Would that really be a physics discovery? I mean I guess everything ultimately is. But it seems like maybe consciousness could be understood in terms of "higher level" sciences - somewhere on the chain of neurology->biology->chemistry->physics.
mmoskal
Consciousness (subjective experience) is possibly orthogonal to intelligence (ability to achieve complex goals). We definitely have a better handle on what intelligence is than consciousness.
tshaddox
That sounds like you’re describing AGI as being impractical to implement in an electronic computer, not impossible in principle.
marcosdumay
> Would that really be a physics discovery?
No, it could be something that proves all of our fundamental mathematics wrong.
The GP just gave the more conservative option.
lll-o-lll
It’s not really “what is the brain doing”; that path leads to “quantum mysticism”. What we lack is a good theoretical framework about complex emergence. More maths in this space please.
Intelligence is an emergent phenomenon; all the interesting stuff happens at the boundary of order and disorder but we don’t have good tools in this space.
sixo
On the contrary, we have one working example of general intelligence (humans) and zero of quantum computing.
null
bee_rider
Do we have a specific enough definition of general intelligence that we can exclude all non-human animals?
jibal
No one said "exclusively humans", and that's not relevant.
mattnewton
Why does it need to exclude all non human animals? Could it not be a difference of degree rather than of kind?
singpolyma3
This makes no sense.
If you believe in eg a mind or soul then maybe it's possible we cannot make AGI.
But if we are purely biological then obviously it's possible to replicate that in principle.
DrewADesign
That doesn’t contradict what they said. We may one day design a biological computing system that is capable of it. We don’t entirely understand how neurons work; it’s reasonable to posit that the differences that many AGI boosters assert don’t matter do matter— just not in ways we’ve discovered yet.
slashdave
We understand how neurons work to quite a bit of detail.
kelnos
I mentioned this in another thread, but I do wonder if we engineer a sort of biological computer, will it really be a computer at all, and not a new kind of life itself?
jibal
It's not "key"; it's not even relevant ... the proof will be in the pudding. Proving a priori that some outcome is possible plays no role in achieving it. And you slid, motte-and-bailey-like, from "know" to "some clear indication of possibility" -- we have extremely clear indications that it's possible, since there's no reason other than a belief in magic to think that "biological" is a necessity.
Whether is feasible or practical or desirable to achieve AGI is another matter, but the OP lays out multiple problem areas to tackle.
slashdave
> We don't know if AGI is even possible outside of a biological construct yet
Of course it is. A brain is just a machine like any other.
root_axis
The practical feasibility of quantum computing is definitely still an open research question.
null
dinobones
Nah,this sounds like a modern remix of Japan’s Fifth Generation Computing project. They thought that by building large databases and with Prolog they would bring upon an AI renaissance.
Just hand waving some “distributed architecture” and trying to duct tape modules together won’t get us any closer to AGI.
The building blocks themselves, the foundation, has to be much better.
Arguably the only building block that LLMs have contributed is that we have better user intent understanding now; a computer can just read text and extract intent from it much better than before. But besides that, the reasoning/search/“memory” are the same building blocks of old, they look very similar to techniques of the past, and that’s because they’re limited by information theory / computer science, not by today’s hardware or systems.
bfung
Yep, the Attention mechanism in the Transformer arch is pretty good.
Probably need another cycle of similar breakthrough in model engineering before this more complex neural network gets a step function better.
Moar data ain’t gonna help. The human brain is the proof: it doesnt need the internet’s worth of data to become good (nor all that much energy).
ComplexSystems
I would like to see what happens if some company devoted your resources to just training a model that is a total beast at math. Feed it a ridiculous amount of functional analysis and machine learning papers, and just make the best model possible for this one task. Then instead of trying to make it cheap so everyone can use it, just set it on the task of figuring out something better than the current architecture and literally have it do nothing else but that and make something based on whatever it figures out. Will it come up with something better than AdamW for optimization? Than transformers for approximating a distribution from a random sample? I don't know, but: what is the point of training any other model?
xyzzy123
Am I the only one who feels that Claude Code is what they would have imagined basic AGI to be like 10 years ago?
It can plan and take actions towards arbitrary goals in a wide variety of mostly text-based domains. It can maintain basic "memory" in text files. It's not smart enough to work on a long time horizon yet, it's not embodied, and it has big gaps in understanding.
But this is basically what I would have expected v1 to look like.
kelnos
> Am I the only one who feels that Claude Code is what they would have imagined basic AGI to be like 10 years ago?
That wouldn't have occurred to me, to be honest. To me, AGI is Data from Star Trek. Or at the very least, Arnold Schwarzenegger's character from The Terminator.
I'm not sure that I'd make sentience a hard requirement for AGI, but I think my general mental fantasy of AGI even includes sentience.
Claude Code is amazing, but I would never mistake it for AGI.
martinald
Totally agree. It even (usually) gets subtle meanings from my often hastily written prompts to fix something.
What really occurs to me is that there is still so much can be done to leverage LLMs with tooling. Just small things in Claude Code (plan mode for example) make the system work so much better than (eg) the update from Sonnet 3.5 to 4.0 in my eyes.
root_axis
The "basic" qualifier is just equivocating away all the reasons why it isn't AGI.
zdragnar
Claude code is neither sentient nor sapient.
I suspect most people envision AGI as at least having sentience. To borrow from Star Trek, the Enterprise's main computer is not at the level of AGI, but Data is.
The biggest thing that is missing (IMHO) is a discrete identity and notion of self. It'll readily assume a role given in a prompt, but lacks any permanence.
atleastoptimal
Any claim of sentience is neither provable nor falsifiable. Caring about its definition has nothing to do with capabilities.
dataviz1000
Student: How do I know I exist?
Philosophy Professor: Who is asking?
Student: I am!
furyofantares
> I suspect most people envision AGI as at least having sentience
I certainly don't. It could be that's necessary but I don't know of any good arguments for (or against) it.
kelseyfrog
Mine is. What evidence would you accept to change your mind?
handfuloflight
Why should it have discrete identity and notion of self?
dsign
I have colleagues that want to plan each task of the software team for the next 12 months. They assume that such a thing is possible, or they want to do it anyway because management tells them to. The first would be an example of human fallibility, and the second would be an example of choosing the path of (perceived) least immediate self-harm after accounting for internal politics.
I doubt very much we will ever build a machine that has perfect knowledge of the future or that can solve each and every “hard” reasoning problem, or that can complete each narrow task in a way we humans like. In other words, it’s not simply a matter of beating benchmarks.
In my mind at least, AGI’s definition is simple: anything that can replace any human employee. That construct is not merely a knowledge and reasoning machine, but also something that has a stake on its own work and that can be inserted in a shared responsibility graph. It has to be able to tell that senior dev “I know planning all the tasks one year in advance is busy-work you don’t want to do, but if you don’t, management will terminate me. So, you better do it, or I’ll hack your email and show everybody your porn subscriptions.”
JSR_FDED
Interesting, I hadn’t thought about it that way. But can a thing on the other end of an API call ever truly have a “stake“?
There is a reason why LLM's are architected the way they are and why thinking is bolted on.
The architecture has to allow for gradient descent to be a viable training strategy, this means no branching (routing is bolted on).
And the training data has to exist, you can't find millions of pages depicting every thought a person went through before writing something. And such data can't exist because most thoughts aren't even language.
Reinforcement learning may seem like the answer here: bruteforce thinking to happen. But it's grossly sample-inefficient with gradient descent and therefore only used for finetuning.
LLM's are regressive models and the configuration that was chosen where every token can only look back allows for very sample-efficient training (one sentence can be dozens of samples).