Training
Loss function
The formula that measures how wrong a model's predictions are during training; training tries to minimise it.
For language models the loss measures how surprised the model is by the actual next token. Lower loss on held-out text usually means a better model.
Frequently asked questions
What does Loss function mean?
The formula that measures how wrong a model's predictions are during training; training tries to minimise it. For language models the loss measures how surprised the model is by the actual next token. Lower loss on held-out text usually means a better model.