Comment by monophonica

Would love to know if you know any other papers like:

Faith and Fate: Limits of Transformers on Compositionality https://arxiv.org/abs/2305.18654

Maybe the analogy is something with gold mining. We could pretend that the machines that mine gold are actually creating gold. Pretending the entire gold mining sector is instead a discovery of alchemy.

Maybe the way alchemy kind of leads to chemistry is the analogy that applies?

I don't even know if that is right though.

The intelligence is in the training data. The model then is extracting the intelligence.

We can't forget Feynman's ideas here that we aren't going to make a robot cheetah that runs fast. We will make a machine that uses wheels. Viewing things through the lense of a cheetah is a category error.

While I agree completely with you we very well both might be completely and utterly wrong. A category error on what intelligence "is".