> I would consider this a death blow paper to the current push for using LLMs and LRMs as the basis for AGI.
Anytime I see "Artificial General Intelligence," "AGI," "ASI," etc., I mentally replace it with "something no one has defined meaningfully."
Or the long version: "something about which no conclusions can be drawn because the proposed definitions lack sufficient precision and completeness."
Or the short versions: "Skippetyboop," "plipnikop," and "zingybang."
Tests designed for humans are not good at testing LLMs because the failure modes are different. Humans don’t have eidetic memories, so they can’t just ingest random facts and recall them at will. Memorizing the relevant facts and figures in a subject and recalling them on a test shows at least some sort of study and likely some sort of overall understanding of the subject. AI, not so much.
A driving test that shows a red octagon with white letters and asks what it means, is a good indicator whether a driver will know that out in the real world, regardless of if it is half covered by a tree, has graffiti on it, is hanging upside down, etc. We found that self driving cars can’t “generalize” like that and failed to recognize a stop sign when it was upside down (and had to be retrained). What is missing is a well functions "judgement" or "reasoning" part that can generalize knowledge, experience, and context. Some models can do something like that for a specific task, but nothing that works long term.
> Probably none.
The qualification is unnecessary; we know the answer is "none". There's a steady stream of lawyers getting penalized for submitting LLM output to judges.
The only definition by which porn, beauty, intelligence, aliveness, and creativity are all known
(seriously, forget this stuff: Consider how you come up with an algorithm for how creative or beautiful something is?)
A line that used to work (back when I was in a part of my life where lines were a thing) was "I can tell you're smart, which means that you can tell I'm smart, because like sees like." Usually got a smile out of 'em.
“The Messiah.” The believers know it’s coming and will transform the world in ways that don’t even make sense to outsiders. They do not change their mind when it doesn’t happen as foreseen.
One vague definition I see tossed around a lot "something can replace almost any human knowledge/white collar worker".
What does that mean in concrete terms? I'm not sure. Many of these models can already pass bar exams but how many can be lawyers? Probably none. What's missing?