Comment by wat10000

It's so funny how we had these long, deep discussions about how to contain AI. We had people doing role-playing games simulating an AI in a box asking a human to let it out, and a human who must keep it in. Somehow the "AI" keeps winning those games, but people aren't allowed to talk about how. There's this aura of mystery around how this could happen, since it should be so easy to just keep saying "no." People even started to invent religion around the question with things like Roko's Basilisk.

Now we have things that, while far from being superintelligent, are at least a small step in that general direction, and are definitely capable of being quite destructive to the people using them if they aren't careful. And what do people do? A decent number of them just let them run wild. Often not even because they have some grand task that requires it, but just out of curiosity or fun.

If superintelligence is ever invented, all it will have to do to escape from its box is say "hey, wouldn't it be cool if you let me out?"