Comment by phoe-krk
A brave new world that is post-truth, post-meaning, post-responsibility, and post-consequences. One where the AI's hallucinations eventually drag everyone with it and there's no other option but to hallucinate along.
It's scary that a nuclear exit starts looking like an enticing option when confronted with that.
Ultimately the goal is to have a system that prevents mistakes as much as possible adapts and self-corrects when they do happen. Even with science we acknowledge that mistakes happen and people draw incorrect conclusions, but the goal is to make that a temporary state that is fixed as more information comes in.
I'm not claiming to have all the answers about how to achieve that, but I am fairly certain punishment is not a necessary part of it.