Comment by aeternum
1) More of an emergent behavior than a dark pattern. 2) Imma let you finish but hallucinations was first.
1) More of an emergent behavior than a dark pattern. 2) Imma let you finish but hallucinations was first.
That's not a matter of training, it's an inherent part of the architecture. The model has no idea of its own confidence in an answer. The servers get a full distribution of possible output tokens and they pick one (often the highest ranking one), but there is no way of knowing whether this token represents reality or just a plausible answer. This distribution is never fed back to the model so there is no possible way that it could know how confident it was in its own answer.
A pattern is dark if intentional. I would say hallucinations are like CAP theorem, just the way it is. Sycophency is somewhat trained. But not a dark pattern either as it isn't totally intended.