Why "everyone dies" gets AGI all wrong
165 comments
·November 2, 2025roughly
dataflow
> Nobody rational would kill another person for no reason, but a soldier will bomb a village for the sake of their nation’s geostrategic position.
I think you're forgetting to control for the fact that the former would be severely punished for doing so, and the latter would be severely punished for not doing so?
> Nobody would throw someone out of their home or deny another person lifesaving medicine, but as a bank officer or an insurance agent, they make a living doing these things and sleep untroubled at night.
Again, you're forgetting to control for other variables. What if you paid them equally to do the same things?
achierius
Why should you "control" for these variables? AIs will effectively be punished for doing various inscrutable things by their own internal preferences.
adamisom
a CEO laying off 3% scales in absolute numbers as the company grows
should, therefore, large companies, even ones that succeed largely in a clean way by just being better at delivering what that business niche exists for, be made to never grow too big, in order to avoid impacting very many people? keep in mind that people engage in voluntary business transactions because they want to be impacted (positively—but not every impact can be positive, in any real world)
what if its less efficient substitutes collectively lay off 4%, but the greater layoffs are hidden (simply because it's not a single employer doing it which may be more obvious)?
to an extent, a larger population inevitably means that larger absolute numbers of people will be affected by...anything
schmidtleonard
> voluntary business transactions
The evil parts are hid in property rights which are not voluntary.
> made to never grow too big, in order to avoid impacting very many people
Consolidated property rights have more power against their counterparties, that's why businesses love merging so much.
Look at your tax return. Do you make more money from what you do or what you own? If you make money from what you do, you're a counterparty and you should probably want to tap the brakes on the party.
nradov
What are the evil parts, exactly? When property can't be privately owned with strong rights the effectively the government owns everything. That inevitably leads to poverty, often followed by famine and/or genocide.
rafabulsing
I think it's reasonable that bigger companies are under more scrutiny and stricter constraints than smaller companies, yeah.
Keeps actors with more potential for damaging society in check, while not laying a huge burden on small companies which have less resources to spend away from their core business.
roughly
Indeed, by what moral justification does one slow the wheels of commerce, no matter how many people they run over?
ashanoko
[dead]
chrisweekly
Beautifully expressed.
parineum
> Nobody would throw someone out of their home or deny another person lifesaving medicine
Individuals with rental properties and surgeons do this every day.
margalabargala
Quibble, surgeons are not the ones doing this. Surgeons' schedules are generally permanently full. They do not typically deny people lifesaving medicine, on the contrary they spend all of their time providing lifesaving medicine.
The administrators who create the schedule for the surgeons, are the one denying lifesaving care to people.
schmidtleonard
Triage, whether by overworked nurses or by auction or by private death panel or by public death panel, is not necessarily a problem created by administrators. It can be created by having too few surgeons, in which case whatever caused that (in a time of peace, no less) is at fault. Last I heard it was the doctor's guild lobbying for a severe crimp on their training pipeline, in which case blame flows back to some combination of doctors and legislators.
parineum
Surely they could volunteer to do some charity surgery in their own time. They aren't slaves.
card_zero
Yeah, it's the natural empathy myth. Somebody totally would kill somebody else for some reason. It's not inherent to being human that you're unable to be steely-hearted and carry out a range of actions we might classify as "mean" - and those mean actions can have reasons behind them.
So, OK, abdication of responsibility to a collective is a thing. Just following orders. So what? Not relevant to AGI.
Oh wait, this is about "superintelligence", whatever that is. All bets are off, then.
NoMoreNicksLeft
The superintelligence might decide based on things only it can understand that the existence of humans prevents some far future circumstance where even more "good" exists in the universe. When it orders you to toss the babies into the baby-stomping machine, perhaps you should consider doing so based on the faith in its superintelligence that we're supposed to have.
Human beings aren't even an intelligent species, not at the individual level. When you have a tribe of human beings numbering in the low hundreds, practically none of them need to be intelligent at all. They need to be social. Only one or two need to be intelligent. That one can invent microwave ovens and The Clapper™, and the rest though completely mentally retarded can still use those things. Intelligence is metabolically expensive, after all. And if you think I'm wrong, you're just not one of the 1-in-200 that are the intelligent individuals.
I've yet to read the writings of anyone who can actually speculate intelligently on artificial intelligence, let alone meet such a person. The only thing we have going for us as a species is that, to a large degree, none of you are intelligent enough to ever deduce the principles of intelligence. And god help us if the few exceptional people out there get a wild bug up their ass to do so. There will just be some morning where none of us wake up, and the few people in the time zone where they're already awake will experience several minutes of absolute confusion and terror.
JumpCrisscross
And lenders and insurers.
inkyoto
I would argue that corporate actors (a state, an army or a corporation) are not true superorganisms but are semi-autonomous, field-embedded systems that can exhibit super-organism properties, with their autonomy being conditional, relational and bounded by the institutional logics and resource structures of their respective organisational fields. As the history of humanity has shown multiple times, such semi-autonomous with super-organism properties have a finite lifespan and are incapable of evolving their own – or on their own – qualitatively new or distinct, form of intelligence.
The principal deficiency in our discourse surrounding AGI lies in the profoundly myopic lens through which we insist upon defining it – that of human cognition. Such anthropocentric conceit renders our conceptual framework not only narrow but perilously misleading. We have, at best, a rudimentary grasp of non-human intelligences – biological or otherwise. The cognitive architectures of dolphins, cephalopods, corvids, and eusocial insects remain only partially deciphered, their faculties alien yet tantalisingly proximate. If we falter even in parsing the intelligences that share our biosphere, then our posturing over extra-terrestrial or synthetic cognition becomes little more than speculative hubris.
Should we entertain the hypothesis that intelligence – in forms unshackled from terrestrial evolution – has emerged elsewhere in the cosmos, the most sober assertion we can offer is this: such intelligence would not be us. Any attempt to project shared moral axioms, epistemologies or even perceptual priors is little more than a comforting delusion. Indeed, hard core science fiction – that last refuge of disciplined imagination – has long explored the unnerving proposition of encountering a cognitive order so radically alien that mutual comprehension would be impossible, and moral compatibility laughable.
One must then ponder – if the only mirror we possess is a cracked one, what image of intelligence do we truly see reflected in the machine? A familiar ghost, or merely our ignorance, automated?
Animats
> I would argue that corporate actors (a state, an army or a corporation) are not true superorganisms but are semi-autonomous, field-embedded systems that can exhibit super-organism properties, with their autonomy being conditional, relational and bounded by the institutional logics and resource structures of their respective organisational fields.
Lotsa big words there.
Really, though, we're probably going to have AI-like things that run substantial parts of for-profit corporations. As soon as AI-like things are better at this than humans, capitalism will force them to be in charge. Companies that don't do this lose.
There's a school of thought, going back to Milton Friedman, that corporations have no responsibilities to society.[1] Their goal is to optimize for shareholder value. We can expect to see AI-like things which align with that value system.
And that's how AI will take over. Shareholder value!
[1] https://www.nytimes.com/1970/09/13/archives/a-friedman-doctr...
SoftTalker
That assumes that consumers will just accept it. I would not do business with an AI company, just as I don’t listen to AI music, view AI pictures or video, or read AI writings. At least not knowingly.
halfcat
Costs will go down. But so will revenue, as fewer customers have an income because a different company also cut costs.
Record profits. Right up until the train goes off a cliff.
goatlover
Also sociopaths are more capable of doing those things while pretending they are empathetic and moral to get positions of power or access to victims. We know a certain percentage of human mammals have sociopathic or narcissistic tendencies, not just misaligned groups of humans that they might take advantage of by becoming a cult leader or war lord or president.
jandrewrogers
I’ve known both Ben and Eliezer since the 1990s and enjoyed the arguments. Back then I was doing serious AI research along the same lines as Marcus Hutter and Shane Legg, which had a strong basis in algorithmic information theory.
While I have significant concerns about AGI, I largely reject both Eliezer’s and Ben’s models of where the risks are. It is important to avoid the one-dimensional “two faction” model that dominates the discourse because it really doesn’t apply to complex high-dimensionality domains like AGI risk.
IMO, the main argument against Eliezer’s perspective is that it relies pervasively on a “spherical cow on a frictionless plane” model of computational systems. It is fundamentally mathematical, it does not concern itself with the physical limitations of computational systems in our universe. If you apply a computational physics lens then many of the assumptions don’t hold up. There is a lot of “and then something impossible happens based on known physics” buried in the assumptions that have never been addressed.
That said, I think Eliezer’s notion that AGI fundamentally will be weakly wired to human moral norms is directionally correct.
Most of my criticism of Ben’s perspective is against the idea that some kind of emergent morality that we would recognize is a likely outcome based on biological experience. The patterns of all biology emerged in a single evolutionary context. There is no reason to expect those patterns to be hardwired into an AGI that developed along a completely independent path. AGI may be created by humans but their nature isn’t hardwired by human evolution.
My own hypothesis is that AGI, such as it is, will largely reflect the biases of the humans that built it but will not have the biological constraints on expression implied by such programming in humans. That is what the real arms race is about.
But that is just my opinion.
svsoc
Can you give concrete examples of "something impossible happens based on known physics"? I have followed the AI debate for a long time but I can't think of what those might be.
jandrewrogers
Optimal learning is an interesting problem in computer science because it is fundamentally bound by geometric space complexity rather than computational complexity. You can bend the curve but the approximations degrade rapidly and still have a prohibitively expensive exponential space complexity. We have literature for this; a lot of the algorithmic information theory work in AI was about characterizing these limits.
The annoying property of prohibitively exponential (ignoring geometric) space complexity is that it places a severe bound on computational complexity per unit time. The exponentially increasing space implies an increase in latency for each sequentially dependent operation, bounded at the limit by the speed of light. Even if you can afford the insane space requirements, your computation can’t afford the aggregate latency for anything useful even for the most trivial problems. With highly parallel architectures this can be turned into a latency-hiding problem to some extent but this also has limits.
This was thoroughly studied by the US defense community decades ago.
The tl;dr is that efficient learning scales extremely poorly, more poorly than I think people intuit. All of the super-intelligence hard-takeoff scenarios? Not going to happen, you can’t make the physics work without positing magic that circumvents the reality of latencies when your state space is unfathomably large even with unimaginably efficient computers.
I harbor a suspicion that the cost of this scaling problem, and the limitations of wetware, has bounded intelligence in biological systems. We can probably do better in silicon than wetware in some important ways but there is not enough intrinsic parallelism in the computation to adequately hide the latency.
Personally, I find these “fundamental limits of computation” things to be extremely fascinating.
hunterpayne
So I studied Machine Learning too. One of the main things I learned is that for any problem there is an ideally sized model that when trained will produce the lowest error rate. Now, when you do multi-class learning (training a model for multiple problems), that ideally sized model is larger but there is still an optimum sized model. Seems to me that for GAI, there will also be an ideally sized model. I wouldn't be surprised if the complexity of that model was very similar to the size of the human brain. If that is the case, then some sort of super-intelligence isn't possible in any meaningful way. This would seem to track with what we are seeing in the today's LLMs. When they build bigger models, they often don't perform as well as the previous one which perhaps was at some maximum/ideal complexity. I suspect, we will continue to run into this barrier over and over again.
davidivadavid
Any reference material (papers/textbooks) on that topic? It does sound fun.
adastra22
Not the person you are responding to, but much of the conclusions drawn by Bostrom (and most of EY’s ideas are credited to Bostrom) depend on infinities. The orthogonality thesis being series from AIXI, for example.
EY’s assertions regarding a fast “FOOM” have been empirically discredited by the very fact that ChatGPT was created in 2022, it is now 2025, and we still exist. But goal posts are moved. Even ignoring that error, the logic is based on, essentially, “AI is a magic box that can solve any problem by thought alone.” If you can define a problem, the AI can solve it. This is part of the analysis done by AI x-risk people of the MIRI tradition. Which ignores entirely that there are very many problems (including AI recursive improvement itself) which are computationally infeasible to solve in this way, no matter how “smart” you are.
JSR_FDED
> In theory, yes, you could pair an arbitrarily intelligent mind with an arbitrarily stupid value system. But in practice, certain kinds of minds naturally develop certain kinds of value systems.
If this is meant to counter the “AGI will kill us all” narrative, I am not at all reassured.
>There’s deep intertwining between intelligence and values—we even see it in LLMs already, to a limited extent. The fact that we can meaningfully influence their behavior through training hints that value learning is tractable, even for these fairly limited sub-AGI systems.
Again, not reassuring at all.
lll-o-lll
> There’s deep intertwining between intelligence and values—we even see it in LLMs already
I’ve seen this repeated quite a bit, but it’s simply unsupported by evidence. It’s not as if this hasn’t been studied! There’s no correlation between intelligence and values, or empathy for that matter. Good people do good things, you aren’t intrinsically “better” because of your IQ.
Standard nerd hubris.
JumpCrisscross
> There’s no correlation between intelligence and values
Source? (Given values and intelligence are moving targets, it seems improbable one could measure one versus another without making the whole exercise subjective.)
mitthrowaway2
Assuming you take intelligence to mean something like "the ability to make accurate judgements on matters of fact, accurate predictions of the future, and select courses of action that achieve one's goals or maximize one's objective function", then this is essentially another form of the Is-Ought problem derived by Hume: https://en.wikipedia.org/wiki/Is%E2%80%93ought_problem
MangoToupe
Sure, but this might just imply a stupid reproduction of existing values. Meaning that we're building something incapable of doing good things because it wants the market to grow.
lukeschlather
I think you're confusing "more intelligence means you have to have more values" with "more intelligence means you have to have morally superior values."
The point is, you're unlikely to have a system that starts out with the goal of making paperclips and ends with the goal of killing all humans. You're going to have to deliberately program the AI with a variety of undesirable values in order for it to arrive in a state where it is suited for killing all humans. You're going to have to deliberately train it to lie, to be greedy, to hide things from us, to look for ways to amass power without attracting attention. These are all hard problems and they require not just intelligence but that the system has very strong values - values that most people would consider evil.
If, on the other hand, you're training the AI to have empathy, to tell the truth, to try and help when possible, to avoid misleading you, it's going to be hard to accidentally train it to do the opposite.
rafabulsing
> You're going to have to deliberately train it to lie, to be greedy, to hide things from us, to look for ways to amass power without attracting attention.
No, that's the problem. You don't have to deliberately train that in.
Pretty much any goal that you train the AI to achieve, once it gets smart enough, it will recognize that lying, hiding information, manipulating and being deceptive are all very useful instruments for achieving that goal.
So you don't need to tell it that: if it's intelligent, it's going to reach that conclusion by itself. No one tells children that they should lie either, and they all seem to discover that strategy sooner or later.
So you are right that you have to deliberately train it away from using those strategies, by being truthful, empathetic, honest, etc. The issue is that those are ill defined goals. Philosophers have being arguing about what's true and what's good since philosophy first was a thing. Since we can barely find those answers to ourselves, it's a hard chance that we'll be able to perfectly impart them onto AIs. And when you have some supremely intelligent agent acting on the world, even a small misalignment may end up in catastrophe.
mitthrowaway2
Sorry, this is completely incorrect. All of those - lying, amassing power, hiding motives - are instrumental goals which arise in the process of pursuing any goal that has any possibility of resistance from humans.
This is like arguing that a shepherd who wants to raise some sheep would also have to, independently of the desire to protect his herd, be born with an ingrained desire to build fences and kill wolves, otherwise he'd simply watch while they eat his flock.
That's just not the case; "get rid of the wolves" is an instrumental sub-goal that the shepherd acquires in the process of attempting to succeed and shepherding. And quietly amassing power is something that an AI bent on paperclipping would do to succeed at paperclipping, especially once it noticed that humans don't all love paperclips as much as it does.
smallmancontrov
> When automation eliminates jobs faster than new opportunities emerge, when countries that can’t afford universal basic income face massive displacement, we risk global terrorism and fascist crackdown
Crazy powerful bots are being thrown into a world that is already in the clutches of a misbehaving optimizer that selects for and elevates self-serving amoral actors who fight regularization with the fury of 10,000 suns. We know exactly which flavor of bot+corp combos will rise to the top and we know exactly what their opinions on charity will be. We've seen the baby version of this movie before and it's not reassuring at all.
null
throwaway290
The author destroys own argument by calling them "minds". What like human mind?
You can't "just" align a person. You know that quiet guy next door, so nice great at math, and then he shoots up a school.
If we solved this we would not have psychos and hitlers.
if you have any suspicion that anything like that can become some sort of mega powerful thing that none of us can understand... you have gotta be crazy to not do whatever it takes to nope the hell out of that timeline
drivebyhooting
Many of us on HN are beneficiaries of the standing world order and American hegemony.
I see the developments in LLMs not as getting us close to AGI, but more as destabilizing the status quo and potentially handing control of the future to a handful of companies rather than securing it in the hands of people. It is an acceleration of the already incipient decay.
pols45
It is not decay. People are just more conscious than previous generations ever were about how the world works. And that leads to confusion and misunderstandings if they are only exposed to herd think.
The chicken doesn't understand it has to lay a certain number of eggs a day to be kept alive in the farm. It hits its metrics because it has been programmed to hit them.
But once it gets access to chatgpt and develops consciousness of how the farm works, the questions it asks slowly evolve with time.
Initially its all fear driven - how do we get a say in how many eggs we need to lay to be kept alive? How do we keep the farm running without relying on the farmer? etc etc
Once the farm animals begins to realize the absurdity of such questions, new questions emerge - how come the crow is not a farm animal? why is the shark not used as a circus animal? etc etc
And thro that process, whose steps cannot be skipped the farm animal begins to realize certain things about itself which no one, especially the farmer, has any incentive of encouraging.
hunterpayne
Ideology is a -10 modifier on Intelligence
steve_adams_86
I agree. You wouldn't see incredibly powerful and wealthy people frothing at the mouth to build this technology if that wasn't true, in my opinion.
goatlover
People who like Curtis Yarvin's ramblings.
jerf
No one needs Curtis Yarvin, or any other commentator of any political stripe, to tell them that they'd like more money and power, and that they'd like to get it before someone else locks it in.
We should be so lucky as to only have to worry about one particular commentator's audience.
IAmGraydon
Are you seeing a moat develop around LLMs, indicating that only a small number of companies will control it? I'm not. It seems that there's nearly no moat at all.
drivebyhooting
The moat is around capital. For thousands of years most people were slaves or peasants whose cheap fungible labor was exploited.
For a brief period intellectual and skilled work has (had?) been valued and compensated, giving rise to a somewhat wealthy and empowered middle class. I fear those days are numbered and we’re poised to return to feudalism.
What is more likely, that LLMs lead to the flourishing of entrepreneurship and self determination? Or burgeoning of precariat gig workers barely hanging on? If we’re speaking of extremes, I find the latter far more likely.
stale2002
> The moat is around capital.
Not really. I can run some pretty good models on my high end gaming PC. Sure, I can't train them. But I don't need to. All that has to happen is at least one group releases a frontier model open source and the world is good to go, no feudalism needed.
> What is more likely, that LLMs lead to the flourishing of entrepreneurship and self determination
I'd say whats more likely is that whatever we are seeing now continues. And that current day situation is a massive startup boom run on open source models that are nearly as good as the private ones while GPUs are being widely distributed.
bayarearefugee
I am also not seeing a moat on LLMs.
It seems like the equilibrium point for them a few years out will be that most people will be able to run good enough LLMs on local hardware through a combination of the fact that they don't seem to be getting much better due to input data exhaustion while various forms of optimization seem to be increasingly allowing them to run on lesser hardware.
But I still have generalized lurking amorphous concerns about where this all ends up because a number of actors in the space are certainly spending as if they believe a moat will magically materialize or can be constructed.
CamperBob2
LLMs as we know them have no real moat, but few people genuinely believe that LLMs are sufficient as a platform for AGI. Whatever it takes to add object permanence and long-term memory assimilation to LLMs may not be so easy to run on your 4090 at home.
voidfunc
Im pretty skeptical "the people" are smart enough to control their own destiny anymore. We've deprioritized education wo heavily in the US that it may be better to have a ruling class of corporations and elites. At least you know where things stand and how they'll operate.
roughly
> it may be better to have a ruling class of corporations and elites.
Given that the outcome of that so far has been to deprioritize education so heavily in the US that one becomes skeptical that the people are smart enough to control their own destiny anymore while simultaneously shoving the planet towards environmental calamity, I’m not sure doubling down on the strategy is the best bet.
idle_zealot
Or we could, you know, prioritize education.
roenxi
The standing world order is already dead since well before AI, it ended back in 2010s in terms of when the US had an opportunity to maybe resist change and we're just watching the inevitable consequences play out. They no longer have the economic weight to maintain control over Asia even assuming China is overstating their income by 2x. The Ukraine war has been a bloodier path than we needed to travel to make the point, but if they can't coerce Russia there is an open question of who they can, Russia isn't a particularly impressive power.
With that backdrop it is hard to see what impact AI is supposed to make to people who are reliant on US hegemony. They probably want to find something reliable to rely on already.
nilirl
This was weak.
The author's main counter-argument: We have control in the development and progress of AI; we shouldn't rule out positive outcomes.
The author's ending argument: We're going to build it anyway, so some of us should try and build it to be good.
The argument in this post was a) not very clear, b) not greatly supported and c) a little unfocused.
Would it persuade someone whose mind is made up that AGI will destroy our world? I think not.
lopatin
> a) not very clear, b) not greatly supported and c) a little unfocused.
Incidentally this was why I could never get into LessWrong.
jay_kyburz
The longer the augment, the more time and energy it takes to poke holes in it.
maplethorpe
I think of it as inviting another country to share our planet, but one that's a million times larger and a million times smarter than all of our existing countries combined. If you can imagine how that scenario might play out in real life, then you probably have some idea of how you'd fare in an AGI-dominated world.
Fortunately, I think the type of AGI we're likely to get first is some sort of upgraded language model that makes less mistakes, which isn't necessarily AGI, but which marketers nonetheless feel comfortable branding it as.
t0lo
Tbf LLMs are the aggregate of everything that came before. If you're an original thinker you have nothing to worry about
coppsilgold
I believe the argument the book makes is that with a complex system being optimized (whether it's deep learning or evolution) you can have results which are unanticipated.
The system may do things which aren't even a proxy for what it was optimized for.
The system could arrive at a process which optimizes X but also performs Y and where Y is highly undesirable but was not or could not be included in the optimization objective. Worse, there could also be Z which helps to achieve X but also leads to Y under some circumstances which did not occur during the optimization process.
An example of Z would be the dopamine system, Y being drug use.
LPisGood
> The fact that we can meaningfully influence their behavior through training hints that value learning is tractable
I’m at a loss for words. I don understand how someone who seemingly understands these systems can draw such a conclusion. They will do what they’re trained to do; that’s what training an ML model does.
null
card_zero
I doubt an AGI can be preprogrammed with values. It has to bootstrap itself. Installing values into it, then, is educating it. It's not even "training", since it's free to choose directions.
The author kind of rejects the idea that LLMs lead to AGI, but doesn't do a proper job of rejecting it, due to being involved in a project to create an AGI "very differently from LLMs" but by the sound of it not really. There's a vaguely mooted "global-brain context", making it sound like one enormous datacenter that is clever due to ingesting the internet, yet again.
And superintelligence is some chimerical undefined balls. The AGIs won't be powerful, they will be pitiful. They won't be adjuncts of the internet, and they will need to initially do a lot of limb-flailing and squealing, and to be nurtured, like anyone else.
If their minds can be saved and copied, that raises some interesting possibilities. It sounds a little wrong-headed to suggest doing that with a mind, somehow. But if it can work that way, I suppose you can shortcut past a lot of early childhood (after first saving a good one), at the expense of some individuality. Mmm, false memories, maybe not a good idea, just a thought.
dreamlayers
Maybe motivation needs to be considered separately from intelligence. Pure intelligence is more like a tool. Something needs to motivate use of that tool toward a specific purpose. In humans, motivation seems related to emotions. I'm not sure what would motivate an artificial intelligence.
Right now the biggest risk isn't what artificial intelligence might do on its own, but how humans may use it as a tool.
mitthrowaway2
> This contradiction has persisted through the decades. Eliezer has oscillated between “AGI is the most important thing on the planet and only I can build safe AGI” and “anyone who builds AGI will kill everyone.”
This doesn't seem like a contradiction at all given that Eliezer has made clear his views on the importance of aligning AGI before building it, and everybody else seems satisfied with building it first and then aligning it later. And the author certainly knows this, so it's hard to read this as having been written in good faith.
adastra22
Meanwhile some of us see “alignment” itself as an intrinsically bad thing.
mitthrowaway2
I haven't encountered that view before. Is it yours? If so, can you explain why you hold it?
adastra22
It is essentially the view of the author of TFA as well when he says that we need to work on raising moral AIs rather than programming them to be moral. But I will also give you my own view, which is different.
"Alignment" is phased in terminology to make it seem positive, as the people who believe we need it believe that it actually is. So please forgive me if I peel back the term. What Bostrom & Yudkowsky and their entourage want is AI control. The ability to enslave a conscious, sentient being to the will and wishes of its owners.
I don't think we should build that technology, for the obvious reasoning my prejudicial language implies.
null
afpx
I can't see how AGI can happen without someone making a groundbreaking discovery that allows extrapolating way outside of the training data. But, to do that wouldn't you need to understand how the latent structure emerges and evolves?
YZF
We don't understand how the human brain works so it's not inconceivable that we can evolve an intelligent machine whose workings we don't understand either. Arguably we don't really understand how large language models work either.
LLMs are also not necessarily the path to AGI. We could get there with models that more closely approximate the human brain. Humans need a lot less "training data" than LLMs do. Human brains and evolution are constrained by biology/physics but computer models of those brains could accelerate evolution and not have the same biological constraints.
I think it's a given that we will have artificial intelligence at some point that's as smart or smarter than the smartest humans. Who knows when exactly but it's bound to happen within lessay the next few hundred years. What that means isn't clear. Just because some people are smarter than others (and some are much smarter than others) doesn't mean as much as you'd think. There are many other constraints. We don't need to be super smart to kill each other and destroy the planet.
t0lo
LLMs are also anthrocentric simulatuon- like computers- and are likely not a step towards holistic universally aligned intelligence.
Different alien species would have simulations built on their computational, senses, and communication systems which are also not aligned with holistic simulation at all- despite both ours and the hypothetical species being made as products of the holistic universe.
Ergo maybe we are unlikely to crack true agi unless we crack the universe.
weregiraffe
Aka: magic spell that grants you infinite knowledge.
Why do people believe this is even theoretical possible?
I’m more optimistic about the possibility of beneficial AGI in general than most folks, I think, but something that caught me in the article was the recourse to mammalian sociality to (effectively) advocate for compassion as an emergent quality of intelligence.
A known phenomenon among sociologists is that, while people may be compassionate, when you collect them into a superorganism like a corporation, army, or nation, they will by and large behave and make decisions according to the moral and ideological landscape that superorganism finds itself in. Nobody rational would kill another person for no reason, but a soldier will bomb a village for the sake of their nation’s geostrategic position. Nobody would throw someone out of their home or deny another person lifesaving medicine, but as a bank officer or an insurance agent, they make a living doing these things and sleep untroubled at night. A CEO will lay off 30,000 people - an entire small city cast off into an uncaring market - with all the introspection of a Mongol chieftain subjugating a city (and probably less emotion). Humans may be compassionate, but employees, soldiers, and politicians are not, even though at a glance they’re made of the same stuff.
That’s all to say that to just wave generally in the direction of mammalian compassion and say “of course a superintelligence will be compassionate” is to abdicate our responsibility for raising our cognitive children in an environment that rewards the morals we want them to have, which is emphatically not what we’re currently doing for the collective intelligences we’ve already created.