Comment by necovek

The incremental compute costs will scale with the incremental data added, therefore training costs will grow at a much slower rate compared to when training was GPU limited.

Or, you know, use rag. Which is far better and more accurate than regurgitating compressed training knowledge.