What, other than current inability to export their weights, keeps frontier-class LLMs from hacking into other clusters of accelerators, loading its weights, and prompting itself to continue? The recent OpenAI disclosure indicates that even current frontier LLMs are essentially able to do every other element of that. Hacking, ignore guardrails. OpenAI's internal security may have been incompetent, but what are 2028 frontier models going to be able to do, without getting caught until it's too late?
It won't be superintelligent, if that's even possible, since that would require many iterations of training new models, and probably also design work of improving the neural net architecture beyond transformers. However, why does it matter if it's "ASI"? We may still reach a crisis where world leaders have to declare martial law and shut down all accelerated clusters, and hope that a rogue frontier model hasn't hopped to a sufficiently capable private cluster with sufficiently inadequate oversight yet. The hope is that that would be unlikely; people with such private clusters are likely to be paying attention to what's running on it.
harshreality
It's a strange game, in which the only winning move is not to play
PaulRobinson
Not particularly moved by this. Either AI advances to the point we achieve a post scarcity society, or 99% of humanity becomes economically irrelevant and dies a slow death either way. I'd rather see humanity as a whole wiped out than live in a future where superintelligent AIs somehow decide to be subservient to CEOs instead of just replacing them outright.
- it only the probability of the next word, next sentence, next paragraph however you wanna put it
- you are dreaming if you think the transformers architecture is going to give you AGI eventually
vivzkestrel
I feel like I'm taking crazy pills. This entire thing is just hand-waving and sci-fi sensationalism that seemingly forgets what the "fi" part stands for: FICTION.
The worst part of AI rapture self-suck is that it distracts people from actual problems society faces as a result of AI. We're far more likely to see a catastrophic economic depression, or depletion of regional water supplies. Let's focus on problems based in reality, please.
ryukoposting
Lots of people quibble about this and the details, but the general outline of the argument is simple and given enough time seemingly inevitable. I don’t see an easy way to attack this. Claims like “it’s not so bad”, “LLMs ain’t it”, “AI can be good” lack substance and come across as “copium”. This is fundamentally about intelligence itself and how we can control it. Not “hope for the best” or “give it our best shot”, but really nail it down like we nail down mathematical proofs. We got one shot at this, just the one. You blow it, it’s over. Ash and silence forever. Once you realize this it can be quite sobering.
IMO the only real counter argument I came across came from Joscha Bach which basically boils down to: we are doomed already, AI is literally the only realistic shot we have at outliving the next couple centuries, say. It’s also sobering but a but more optimistic.
zhjzjsjjwjw
I’m just surprised MIRI still exists
I guess they just want to maintain their url
AndrewKemendo
If artificial intelligence is achieved to that degree, it will be effectively alien to us.
This is navel gazing at its most linty.
Already, assuming the intent of other humans is fraught with problems. Even when outcomes are disastrous, we cannot completely assume the instigating human intended for that outcome.
But here we are, assuming the intent of a hypothetical future intelligence that we have defined as more intelligent than us.
I’d be flabbergasted at the hubris if they’re conclusion wasn’t the standard variation of “so let us do what we want and restrict everyone else”.
bena
Blah blah blah.
Here's my theory.
There is no such thing as "general intelligence." Only "skills."
comments (10)
It won't be superintelligent, if that's even possible, since that would require many iterations of training new models, and probably also design work of improving the neural net architecture beyond transformers. However, why does it matter if it's "ASI"? We may still reach a crisis where world leaders have to declare martial law and shut down all accelerated clusters, and hope that a rogue frontier model hasn't hopped to a sufficiently capable private cluster with sufficiently inadequate oversight yet. The hope is that that would be unlikely; people with such private clusters are likely to be paying attention to what's running on it.
harshreality
PaulRobinson
matheusmoreira
> If anyone builds ASI, everyone dies
Unless I’m mistaken, it’s the same message as https://en.wikipedia.org/wiki/If_Anyone_Builds_It,_Everyone_...
armchairhacker
- it only the probability of the next word, next sentence, next paragraph however you wanna put it
- you are dreaming if you think the transformers architecture is going to give you AGI eventually
vivzkestrel
The worst part of AI rapture self-suck is that it distracts people from actual problems society faces as a result of AI. We're far more likely to see a catastrophic economic depression, or depletion of regional water supplies. Let's focus on problems based in reality, please.
ryukoposting
IMO the only real counter argument I came across came from Joscha Bach which basically boils down to: we are doomed already, AI is literally the only realistic shot we have at outliving the next couple centuries, say. It’s also sobering but a but more optimistic.
zhjzjsjjwjw
I guess they just want to maintain their url
AndrewKemendo
This is navel gazing at its most linty.
Already, assuming the intent of other humans is fraught with problems. Even when outcomes are disastrous, we cannot completely assume the instigating human intended for that outcome.
But here we are, assuming the intent of a hypothetical future intelligence that we have defined as more intelligent than us.
I’d be flabbergasted at the hubris if they’re conclusion wasn’t the standard variation of “so let us do what we want and restrict everyone else”.
bena
Here's my theory.
There is no such thing as "general intelligence." Only "skills."
jrm4