Comment by ijustlovemath 4 days ago
> Astonishingly, this approach, done correctly, produces better forecasts than traditional simulations of the physics of our atmosphere.
It seems like this is another instance of The Bitter Lesson, no?
Thank you - I hadn't heard of it before. It seems to have parallels with LLMs - our most general intelligent systems have come from producing a workable architecture for what seems to be the bare minimum for communicating intelligence while also having plenty of training data (language), then simply scaling up.
I thought this was a good quote:
> We want AI agents that can discover like we can, not which contain what we have discovered.
I'm not sure I buy The Bitter Lesson, tbh.
Deep Blue wasn't a brute-force search. It did rely on heuristics and human knowledge of the domain to prune search paths. We've always known we could brute-force search the entire space but weren't satisfied with waiting until the heat death of the universe for the chance at an answer.
The advances in machine learning do use various heuristics and techniques to solve particular engineering challenges in order to solve more general problems. It hasn't all come down to Moore's Law.. which stopped bearing large fruit some time ago.
However that still comes at a cost. It requires a lot of GPUs, land, energy, and fresh water, and Freon for cooling. We'd prefer to use less of these resources if possible while still getting answers in a reasonable amount of time.
Deep blue had to use the techniques it did due to the limitations of the hardware of the time. Deep blue would almost certainly lose against AlphaZero, even if you tuned it to modern hardware. All you have to do 'manually' is teach it the rules/give it a loss function, then you just let it do its thing.
It's certainly true that "just throw a bunch of GPUs at it" is wasteful, but it does achieve results.
Certainly does! We’ve had expert systems and various AI techniques for decades that weren’t efficient enough to run even though theoretically they would yield answers.
And even though solutions to many such problems were in the NP or NP-hard categories it didn’t mean that we couldn’t get useful results.
But it still gave us better results by applying what we know about search strategies and reinforcement to provide guidance and heuristics. Even Alpha didn’t use the most general algorithms and throw hardware at the problem. Still took quite a lot of specialized software and methods to fine-tune the overall system to produce the results we want.
Today's best chess models use no heuristics, I think starting with stockfish 16 they got rid of HCE (hand crafted evaluation), they're now neural nets and would absolutely eat Deep Blue
That's a highly controversial claim that would need a whole host of published peer-reviewed research papers to support it. Physics-based simulations (initial state input, then evolve according to physics applied to grids) have improved but not really because of smaller grids, but rather by running several dozen different models and then providing the average (and the degree of convergence) as the forecast.
Notably forecast skill is quantifiable, so we'd need to see a whole lot of forecast predictions using what is essentially the stochastic modelling (historical data) approach. Given the climate is steadily warming with all that implies in terms of water vapor feedback etc., it's reasonable to assume that historical data isn't that great a guide to future behavior, e.g. when you start having 'once every 500 year' floods every decade, that means the past is not a good guide to the future.
Given 50 states and independent weather in each state, on average one state would experience each "once every 500 years" extreme weather event every decade. Of course in reality weather is not independent across political borders, but there are also many more locations where flood levels can be measured than just one per state. So depending on the details "once every 500 years" may not be as rare as it sounds, even without deviation from historical patterns.
Yes, it seems like it. Although I would imagine the features and architecture of the model still take some physics into account. You can't just feed weather data into an LLM, after all.
I was surprised: https://arxiv.org/abs/1912.09363
It's not exactly an LLM but it works in a similar fashion.
For anyone else who's also in today's lucky 10,000: http://www.incompleteideas.net/IncIdeas/BitterLesson.html