Comment by Davidzheng

but natural brains sleep too, which I guess is your point. But actually is it even clear in human brains whether most of neural compute is evaluation vs training? maybe the brain is like for e.g. capable of running 20T model of compute and deploying like 2B model at given time and most of compute is training in background new models--I mean like you say we have no idea except for training from scratch, but if we are working much below capacity of compute we could actually actively train from scratch repeatedly (like the xAI cluster could probably train gpt4o size in a matter of hours)