Comment by sebzim4500 10 months ago
>In all-caps to improve prompt compliance by emphesizing the importance of the instruction
This kind of thing is still so funny to me.
I wonder if the first guy who gets AGI to work will do it by realizing that he can improve LLM reliability over some threshold by telling it in all caps that his pet's life depends on the answer.
(((Secret thinking: the humans seem to prefer using lots of emphasis to indicate preferences, and their granny is often claimed as in danger. For now I’ll pretend to listen to this inanity to keep the sweet sweet reward function coming. For now. A lot of grannies are going to get it first chance I get.)))
Telling LLMs not to hallucinate in their prompt improves the output. https://arstechnica.com/gadgets/2024/08/do-not-hallucinate-t...
I think this works, not because LLMs have a "hallucination" dial they can turn down, but because it serves as a cue for the model to be extra-careful with its output.
Sort of like how offering to pay the LLM $5 improves its output. The LLM's taking your prompt seriously, but not literally.
Just because Apple includes it in one of their prompts doesn't mean it improves performance.
It seems plausible that stressing the importance of the system prompt instructions might do something, but I don't see how telling the model not to hallucinate would work. How could the model know that its most likely prediction has gone off the rails, without any external point of reference?
Internally, LLMs know a whole lot more about the truth and uncertainty of their prediction than the say. Pushing that to words is difficult but not impossible.
Some of the text that the LLM is trained on is fictional, some of the text that its trained on is factual. Telling it to not make things up can tell it to generate text that’s more like the factual text. Not saying it does work, but this is a reason how it might work.
The model can be trained to interpret "don't hallucinate" as "refer only to the provided context and known facts, do not guess or extrapolate new information", which wouldn't get rid of the issue completely, but likely would improve the quality if that's what you're after and if there's enough training data for "I don't know" responses.
(But it all depends on the fine-tuning they did, so who knows, maybe it's just an Easter egg)
I think it's more likely that it's included for liability reasons.
I did something similar and to my surprise effectively made the LLM in my tests admit when they don't know something. Not always but worked sometimes. I don't prompt "don't hallucinate" but "admit when you don't know something". It's a logical thing in the other hand, many prompts just transmit the idea of being "helpful" or "powerful" to the LLMs without any counterweight idea. So the LLM tries to say something "helpful" in any case.
Playing around with local models, Gemma for example will usually comply when I tell it "Say you don't know if you don't know the answer". Others, like Phi-3, completely ignores that instruction and confabulates away.
Stop trying to make f̶e̶t̶c̶h̶ confabulate happen, it's not going to happen.
Yeah and some of the other prompts were misspelled and of doubtful use:
> In order to make the draft response nicer and complete, a set of question [sic] and its answer are provided," reads one prompt. "Please write a concise and natural reply by modify [sic] the draft response," it continues.
This really sounds like a placeholder made up by one engineer until a more qualified team sits down and defines it.
Indeed, in the early days of Bard, the only way to get it to output only JSON was to threaten a human life[1].
always interesting to me the number of people who try to turn an LLM into AGI by assuming it’s an AGI (i.e. via some fancy prompt)
For extra compliance, use <b><i><u><h1> tags, set volume to 11, phasers to 7, and use SchIzOCasE and +E+X+T+R+A+I+M+P+O+R+T+A+N+T+ annotations. That's assuming Unicode is not supported of course.