I totally agree with most of the article, but the hallucinations bit puzzles me. If it’s genuinely an unchangeable limitation of the product (as hallucinations are with LLMs) it’s good to set the right expectation rather than making promises you can’t deliver on.
The LLM doesn't produce erroneous output, its generated tokens fall within the statistical limits of the preset configuration, unless some kind of bitflip messed up the model in memory somehow. An LLM doesn't tell the truth or answer a question, it just spits out tokens. Its training doesn't involve validating whether or not the output forms a true fact or statement, but rather if the output looks like one. For the same reason, an LLM cannot lie, because an LLM doesn't have any intention, nor can it tell the truth. That level of thinking is beyond the capability of an LLM.
The term "hallucinations" are an anthropomorphised interpretation of valid output that's factually incorrect. It happens to people all the time (the human brain will make up any missing memories and subconsciously explain away inconsistencies, which only becomes obvious once you're dealing with someone with memory problems), so it feels like a decent term to use for garbage information produced without any ill intent.
The problem lies with the AI companies convincing their customers that the output generated by their tools is probable to mean anything. Probability engines are sold as some kind of chat program or even as some kind of autonomous agent because the output comes close enough to pass the Turing test to most people. LLMs can only mimic intelligence, interactivity, or any other kind of behavior, they cannot actually think or reason.
If people knew what they were operating, the "hallucinations" wouldn't be a problem. Unfortunately, that would take out most of the confidence people have in these tools, so you won't see the AI salesmen provide their customers with reasonable expectations.
If I write JavaScript that outputs “5+5=55” we would rightfully call that an error regardless of the implementation details. But when an LLM does it, you’re saying it’s not an error, it’s “generated tokens that fall within the statistical limits of the present configuration.”
My point was that, from the perspective of the end user, this is an error. If ChatGPT was described as a “random text generator” then sure maybe this wouldn’t be considered an error because “5+5=55” is random text. But that’s not what 90% or ChatGPT users expect nor how the app is marketed, it’s marketed as an Artificial Intelligence Assistant.
Exactly, it’s like getting mad at Isaac Newton because the root finding algorithm doesn’t work on particular functions or something. Like, the issue is not the algorithm, it’s that your expectations were incorrect. This is not an all-knowing machine, it’s a “what would a human say” estimation function.
Similarly, anyone who claims that LLMs in their current form are going to achieve AGI sounds like Newton bragging that he had solved all of math.
> It doesn't matter to the end user if hallucinations are an unchangeable limitation
Of course it does. I don’t go around complaining that my stove burns me when I touch it. Anyone who knows anything about LLMs at this point knows not to do anything mission critical with them and that’s a good thing.
You are not allowed to tell the truth about LLMs, it is simply outside of the current Overton window. In a year or two, this will be retconned. I guarantee it.
It doesn't matter to the end user if hallucinations are an unchangeable limitation, the fact that they happen undermines the confidence that people have in them as a tool.
I've wondered the same thing as the author about why we even call them "hallucinations." They're errors, the LLM generated an erroneous output.