Comment by someothherguyy
Comment by someothherguyy 6 days ago
its vague, and could have meant anything. everyone knew parameters would grow and its reasonable to expect that things that grow have diminishing returns at some point. this happened in late 2023 and throughout 2024 as well.
That quote almost perfectly describes o1, which was the first major model to explicitly build in compute time as a part of its scaling. (And despite claims of vagueness, I can't think of a single model release it describes better). The idea of a scratchpad was obvious, but no major chatbot had integrated it until then, because they were all focused on parameter scaling. o1 was released at the very end of 2024.