Training
Gradient descent
The optimisation method that nudges a model's parameters, step by step, in the direction that most reduces its loss.
Every large model is trained with some variant of gradient descent, repeated over trillions of tokens. The 'learning rate' controls how big each step is.
Frequently asked questions
What does Gradient descent mean?
The optimisation method that nudges a model's parameters, step by step, in the direction that most reduces its loss. Every large model is trained with some variant of gradient descent, repeated over trillions of tokens. The 'learning rate' controls how big each step is.