Comment by voidspark 2 days ago
Chollet's argument was that it's not "true" generalization, which would be at the level of human cognition. He sets the bar so high that it becomes a No True Scotsman fallacy. The deep neural networks are practically generalizing well enough to solve many tasks better than humans.
He is setting a bar, saying that that is the "true" generalization.
Deep neural networks are definitely performing generalization at a certain level that beats humans at translation or Go, just not at his ARC bar. He may not think it's good enough, but it's still generalization whether he likes it or not.
I'm not convinced either of your examples is generalization. Consider Go. I don't consider a procedural chess engine to be "generalized" in any sense yet a decent one can easily beat any human. Why then should Go be different?
A procedural chess engine does not perform generalization, in ML terms. That is an explicitly programmed algorithm.
Generalization has a specific meaning in the context of machine learning.
The AlphaGo Zero model learned advanced strategies of the game, starting with only the basic rules of the game, without being programmed explicitly. That is generalization.
No. His argument is definitely closer to LLMs can't generalize. I think you would benefit from re-reading the paper. The point is that a puzzle consisting of simple reasoning about simple priors should be a fairly low bar for "intelligence" (necessary but not sufficient). LLMs performs abysmally because they have a very specific purpose trained goal that is different from solving the ARC puzzles. Humans solve these easily. And committees of humans do so perfectly. If LLMs were intelligent they would be able to construct algorithms consisting of simple applications of the priors.
Training to a specific task and getting better is completely orthogonal to generalized search and application of priors. Humans do a mix of both search of the operations and pattern matching of recognizing the difference between start and stop state. That is because their "algorithm" is so general purpose. And we have very little idea how the two are combined efficiently.
At least this is how I interpreted the paper.