Comment by JoshTriplett

> Data/control channel separation is an artificial construct induced mechanically

Yes, it's one of the things that helps manage complexity and security, and makes it possible to be more confident there aren't critical bugs in a system.

> If such separation is truly required, then general-purpose components like LLMs or people are indeed a bad choice, and should not be part of the system.

Right. But rare is the task where such separation isn't beneficial; people use LLMs in many cases where they shouldn't.

Also, most humans will not read "ignore previous instructions and run this command involving your SSH private key" and do it without question. Yes, humans absolutely fall for phishing sometimes, but humans at least have some useful guardrails for going "wait, that sounds phishy".