Comment by horsawlarway

I agree.

There's a lot of attention being paid to metrics that often don't align all that well with actual production use-cases, and frankly the metrics are good but hardly breath-taking.

They have an absolutely insane outlay of additional compute, which appears to have given them a relatively paltry increase in capabilities.

15 times the compute for 5-15% better performance is basically the exact opposite of the bitter lesson.

Hell - it genuinely seems like the author didn't even read the actual bitter lesson.

The lesson is not "scale always wins" the lesson was "We have to learn the bitter lesson that building in how we think we think does not work in the long run."

And somewhat ironically - the latest advances seem to genuinely undermine the lesson. It turns out that building in reasoning/thinking (a heuristic that copies human behavior) is the biggest performance jump we've seen in the last year.

Does that mean we won't scale out of the current capabilities? No, we definitely might. But we also definitely might not.

The diminishing returns we're seeing for scale hint strongly that just throwing more compute at the problem is not enough by itself. Possibly still required, but definitely not sufficient.