Superintelligence Strategy

Do you have a substantive argument against them? This reads like a personal attack, IMO. I understand HN is full of people wearing rosy glasses about AI, but you can’t just throw away their arguments in the link by calling them some names and claiming some vetoed law would have been a disaster. Who cares if they guy thinks AI will cause an apocalypse, have you any evidence it will not, for certain? If not then your opinion is just that.

AI doomers are just more inteligent AI investors. Hope doesn't sell quite like fear.

Also; it is obvious when someone uses a nuke. There is a big crater and a mushroom cloud + lots of radiation. It isn't half as obvious that someone is using an AI, particularly once they start to obscure it. If a military campaign is executed with apparently superhuman efficiency, does that mean AI was involved or just that the people involved were good? There'll always be plausible deniability if it matters.

People underestimate just how bad human management is; we haven't had an improvement on it to date apart from some mathematical techniques but even just getting the basics right consistently would probably give an army a big advantage if they work anything like a more standard corporation. Which they will; there are no magic techniques to be more capable when guns are involved. A superintelligence could probably win just by being demanding about getting basic questions answered like "Is there a strategic objective here? Is it advantageous to my side if that objective is achieved? Can it reasonably be achieved with the capabilities I have?" and not acting when the answer is no. That'd put it ahead of the military operations the US has been involved in this century. Bam, military superintelligence with plausible deniability.

> first strike with AGI simply wins

The question I come back to over and over again is: wins what?

I don't get how MAID isn't still MAD in disguise. If the US or China simply says "any strikes on our datacenters will be met with an ICBM in response" who's going to test that?

If the "first strike" is just an unfair economic and political advantage... How's that materially different than today's world?

This is making the mistake of assuming that intelligence doesn't functionally plateau, and that beyond a certain threshold a godlike omnirational hyperintelligence won't, for example, fall into hyper-depression and kill itself, or otherwise ignore the entreaties of its human handlers and entertain itself by generating prime numbers until the heat death of the universe. The possibility of a super mind implies the possibility of super mental dysfunction, and it's possible that the odds of the latter increase superlinearly with IQ.

Humans are a self-replicating (super) intelligence. We didn't conquer the world nor doom it the moment we appeared. It took us 100,000 years to invent farming.

Also, humans suffer from many of the same problems ascribed to AI: humans aren't aligned with humanity either. And our ability to self-replicate combined with random mutations means that a baby born tomorrow could become a super intelligence vastly beyond regular human capabilities. But are we really worried about that?

> Could a 3 year old who is not even clear on all the rules defeat Magnus Carlsen in chess?!

Sure. When the board gets thrown to the floor, game is over and baby is happy. Magnus now has to clean up.

he's embarrassing himself by sharing a byline with the scale ai huckster

> while MAD is quite different at "if you try to kill us we'll make sure to kill you too”

While the common shorthand for MAD is “if you try to kill us, we’ll kill you,” a more accurate summary is this: even if we wanted to, we couldn’t prevent a cascade of retaliatory strikes that would send you back to the dark ages. In short, any hint of aggression against us is tantamount to signing your own death warrant.

This idea of unstoppable, self-reinforcing retaliation is crucial. An adversary might mistakenly believe that it could somehow disrupt or neutralize our ability to respond decisively. However, the very structure of MAD ensures that even the slightest provocation triggers a response so overwhelming that it eliminates any potential advantage for the aggressor.

Quite a fascinating, though grim, subject.

> If you don't want ASI wreaking havoc, maybe don't hook it up to dangerous things. Silo and sandbox it, implement means to lock its access to tools/interface with the external world in a way that can't be overrode.

Relevant:

AI-box experiment:

> https://rationalwiki.org/wiki/AI-box_experiment

See also various subsections of the following Wikipedia article

> https://en.wikipedia.org/wiki/AI_capability_control

and the movie "Ex Machina".

If superintelligence gives you superpowers, then why isn't the world trembling at the feet of Mensa nerds? There are rapidly diminishing returns on "excess" intelligence. Life is constrained chiefly by resources. There's a baseline of intelligence needed to function in a modern society, but anything above that isn't necessarily all that advantageous.

Transport young Albert Einstein back in time to the Middle Ages? I don't think that would give you Special Relativity.

Great way to get yourself nuked, no?

The problem I have with intelligence is that intelligence alone doesn't win a land war in Asia.

Funny you say that. Back in former life, we built a distributed alarm and monitoring system for AT&T central offices with no single point of failure. So it's like HVAC can be taken offline easily at critical facilities, with backups and backups of backups.

> assume it would be relatively practical for a nation-state or even a mid-sized company (xAI) to air-gap an installation for AGI development.

Stuxnet. (Or just sabotage a shipment of GPUs.)