Comment by matheusd
How about this for an evaluation: Have this (trained-on-older-corpus) LLM propose experiments. We "play the role of nature" and inform it of the results of the experiments. It can then try to deduce the natural laws.
If we did this (to a good enough level of detail), would it be able to derive relativity? How large of an AI model would it have to be to successfully derive relativity (if it only had access to everything published up to 1904)?
I don't know if any dataset of pre 1904 writing would be large enough to train a model that would be smart enough. I suspect that current sized SOTA models would at least get to special relativity, but for general relativity and quantum mechanics I am less sure.