Comment by TheOtherHobbes

Our thousands of years of experience building systems from humans have created systems that are really not that great in terms of security, survivability, and stability.

With AI of any kind you're always going to have the problem that a black hat AI can be used to improvise new exploits - > Red Queen scenario.

And training a black hat AI is likely immensely cheaper than training a general LLM.

LLMs are very much not just regular narrow-domain computer programs. They're a structural issue in the way that most software - including cloud storage/processing - isn't.