Comment by fragmede 4 days ago
But the same way you bootstrap a new compiler from stage 1 to stage 2 and self hosted, LLMs have advanced to the point that they can be used on its training data to decide if, eg the Earth is actually flat or not.
Then a very important first question is how do we (humans) discern facts in such cases?
I was rather explicit about that, you memorize them from trusted sources (or directly observe them). There's no question. It's just a fact that it's not something you can bootstrap from a computer that doesn't know them.
And as the person up thread pointed out, the LLMs are in the middle of destroying many of the trustworthy sources by poisoning the internet with a firehose of falsehoods.
It's all about trust. How do we help machines (and humans) know what to trust?
Are you saying human brain is kind of similarly vulnerable to well-crafted facts? Does it mean any intelligence (human or non-human) needs a large amount of generally factual data to discern facts from fakes, which is an argument toward AIs that can accumulate huge swath of factual data?
I feel like you're trying to twist my words into something they don't resemble at all.
I'm not saying anything is vulnerable to anything. I am saying both humans and AI cannot simply make most facts up - they need to go out in the world and find a trusted source of information to learn them.
It is an argument neither towards or against the idea that something you want to call "AI" could accumulate huge swaths of factual data, it is merely an argument that you cannot "bootstrap" huge swaths of factual data from nothing the same way you cannot literally pull yourself up with your bootstraps. If you want the information, you have to collect it from the environment.
The difference that a compiler is (generally) deterministic. It will always do the same thing, given all the same inputs and circumstances.
An LLM is not, it's probabilistic text. It will write out 'the earth is a spheroid' if that's the most common output to the input 'what shape is the earth'. But it does not understand what it is writing. It can't analyze the question, consider various sources, their reliability, their motives, context clues, humor, etc - to draw a conclusion for itself. It can't make a mistake and then learn from that mistake when corrected.
probabilistically, why does that matter? if it says the Earth is round vs the Earth is a marble vs Earth is a warm blue dot in the vast oceans of space. Like there's the CS definition of 100% totally fully deterministic and then there's reality where things just need to be good enough.
What if 0.5% of the time it says that the Earth is flat? Being used millions of times per day, it will tell thousands of people that the earth is actually flat, and may convince some of them of this false fact.
Whatever shows up most commonly in the training data is is what an LLM will output. It's more complicated than that of course, but that's the basic idea.
And I think you missed the point. If you knew which were 'correct' and which were 'incorrect' then you could avoid the problem altogether. But that would mean someone would have to curate the entire internet, looking for anything that's 'incorrect' (or intended as humor) and making sure it doesn't end up in the training data Or LLM-generated content, to avoid cascading failures.
That's an unbelievable amount of work. It's essentially impossible, no matter how much money you throw at it. There's so much content being made every day you couldn't even keep up with what's being added let alone what's already there.
> Whatever shows up most commonly in the training data is is what an LLM will output. It's more complicated than that of course, but that's the basic idea.
The most common thing in the training data is the letter 'e'. If you're going to explain how an LLM works it needs to explain why it's able to form sentences at all.
In particular answering questions is a behavior which only appears after posttraining, and the posttraining objective has absolutely nothing to do with what's "most common" in the pretraining data.
> But that would mean someone would have to curate the entire internet, looking for anything that's 'incorrect' (or intended as humor) and making sure it doesn't end up in the training data
Show the LLM the source URL during pretraining so it can cluster them together.
https://arxiv.org/abs/2501.01956
The cheap version of this technique is to find trustworthy text (Wikipedia, answers you paid people to write, high upvoted Reddit comments) and train on it more than once. The rest falls out through emergent magic (reliable sources have different writing styles than unreliable ones and RL points it to the part of latent space with the reliable sources, or something.)
Besides that, if it encounters 95%/5% right/wrong answers to some question during training, that will have a different effect than 100%/0%. It does know when something is debated.
Most facts about the world can't be deduced from logic. They're just facts, to memorize. The King's lefthanded. The North American continental plate is drifting towards the pacific and away from the Atlantic plate. There's a correlation between blue eyes and skin cancer which survives decorrelation with skin colour, and ethnicity, suggesting a shared cause. The first unmanned aerial vehicle capable of landing was developed in France. A general named Rogers led the British in the war of 1812.
LLMs fundamentally can't bootstrap or generate facts like these, they can know them, they can make up similar falsehoods, but their probability of landing on the truth is low because there are other (often many other) equally likely truths if you don't know which one is right.
(Please note: I made up all the "facts" in this post)