Comment by MikeTheGreat

Ah. Thanks for posting - this makes a lot of sense.

I can totally see how they're able to pre-train models no problem, but are having trouble with the "noticeably better" part.

Thanks!