Comment by cocogoatmain

Want to also add that the model doesn’t know how to respond in a user-> assistant style conversation after it’s pretraining, and it’s a pure text predictor (look at the open source base models)

There’s also what is being called mid-training where the model is trained on high(er) quality traces and acts as a bridge between pre and post training