Comment by majormajor

It's not claimed to be an "objectively optimal" solution, it's claimed to represent how the world works.

I don't know where you're going with discussion of destruction and killing, but even fairly simple consumer products have any number of edge cases that initial specifications rarely capture. I'm not sure what "objectively optimal" is supposed to mean here, either.

If a spec described every edge case it would basically be executable already.

The pain of developing software at scale is that you're creating the blueprint on the fly from high-level vague directions.

Something trivial that nevertheless often results in meetings and debate in the development world:

Spec requirement 1: "Give new users a 10% discount, but only if they haven't purchased in the last year."

Spec requirement 2, a year later: "Now offer a second product the user can purchase."

Does the 10% discount apply to the second product too? Do you get the 10% discount on the second product if you purchased the first product in the last year, or does a purchase on any product consume the discount eligibility? What if the prices are very different and customers would be pissed off if a $1 discount on the cheaper product (which didn't meet their needs in the end) prevented them from getting a 10$ discount 9 months later (which they think will)? What if the second product is a superset of the first product? What if there are different relevant laws in different jurisdictions where you're selling your product?

Agents aren't going to figure out the intent of the company's principal's automatically here because the decision maker doesn't actually even realize it's a question until the implementers get into the weeds.

A sufficiently advanced agent would present all the options to the person running the task, and then the humans could decide. But then you've slowed things back down the pace of the human decision makers.

The complexities only increase as the product grows. And once you get into distributed or concurrent systems even most of our code today is ambiguous enough about intent that bugs are common.