Comment by no_wizard

> If I say "well I suppose you'd at least need to do A to qualify" and then later say "huh I guess A wasn't sufficient, looks like you'll also need B" that is not shifting the goalposts.

I dunno, that seems like a pretty good distillation of what moving the goalposts is.

> I’ll know it when I see it, and I haven’t seen it. QED

While pithily put, thats not a compelling argument. You feel that LLMs are not intelligent. I feel that they may be intelligent. Without a decent definition of what intelligence is, the entire argument is silly.

Why presuppose that it's possible to test intelligence via text? Most humans have been illiterate for most of human history.

I don't mean to claim that it isn't possible, just that I'm not clear why we should assume that it is or that there would be an obvious way of going about it.

Or maybe accept that LLMs are intelligent and it's human bias that is the oddity here.

> problem-solving strategies and an execution engine

Extremely unfounded claims. See: the root comment of this tree.

LLMs are completely deterministic. Their fundamental output is a vector representing a probability distribution of the next token given the model weights and context. Given the same inputs an identical output vector will be produced 100% of the time.

This fact is relied upon by for example https://bellard.org/ts_zip/ a lossless compression system that would not work if LLMs were nondeterministic.

In practice most LLM systems use this distribution (along with a “temperature” multiplier) to make a weighted random choice among the tokens, giving the illusion of nondeterminism. But there’s no fundamental reason you couldn’t for example always choose the most likely token, yielding totally deterministic output.

This is an excellent and accessible series going over how transformer systems work if you want to learn more. https://youtu.be/wjZofJX0v4M

The LLM's output is chaotic relative to the input, but it's deterministic right? Same settings, same model, same input, .. same output? Where does the chain get broken here?

The only reason LLMs are stochastic instead of deterministic is a random number generator. There is nothing inherently non-deterministic about LLM algorithms unless you turn up the "temperature" of selecting the next word. The fact that determinism can be changed by turning a knob is clear evidence that they are closer to a database or search engine than a human.

> A lot of people would say that LLMs generalize pretty well

What do you mean here? The trained model, the inference engine, is the one that makes an LLM for "a lot of people".

> they certainly can understand natural language sequences that are not present in their training data

Keeping the trained model as LLM in mind, I think learning a language includes generalization and is typically achieved by a human, so I'll try to formulate:

Can a trained LLM model learn languages that hasn't been in its training set just by chatting/prompting? Given that any Korean texts were excluded from the training set, could Korean be learned? Does that even work with languages descending from the same language family (Spanish in the training set but Italian should be learned)?

No. His argument is definitely closer to LLMs can't generalize. I think you would benefit from re-reading the paper. The point is that a puzzle consisting of simple reasoning about simple priors should be a fairly low bar for "intelligence" (necessary but not sufficient). LLMs performs abysmally because they have a very specific purpose trained goal that is different from solving the ARC puzzles. Humans solve these easily. And committees of humans do so perfectly. If LLMs were intelligent they would be able to construct algorithms consisting of simple applications of the priors.

Training to a specific task and getting better is completely orthogonal to generalized search and application of priors. Humans do a mix of both search of the operations and pattern matching of recognizing the difference between start and stop state. That is because their "algorithm" is so general purpose. And we have very little idea how the two are combined efficiently.

At least this is how I interpreted the paper.