Training
Pre-training
The first, largest stage of training, where a model learns language and world knowledge by predicting text across a huge dataset.
Pre-training produces a capable but raw model. Later stages, fine-tuning and reinforcement learning, turn it into an assistant that follows instructions and behaves safely.
Frequently asked questions
What does Pre-training mean?
The first, largest stage of training, where a model learns language and world knowledge by predicting text across a huge dataset. Pre-training produces a capable but raw model. Later stages, fine-tuning and reinforcement learning, turn it into an assistant that follows instructions and behaves safely.