Comment by robertlagrant

The cause is easy: the people specifying the problem to be solved don't ask for an independent set of tests that are human-readable and can be run automatically to verify the system.

If they did that, then the system would be verifiable and so would changes to it - the tests would simply need to be adapted to talk to the new version of the system.

Too late now, of course. But that's what should be done.