Inference
Sampling
How a model picks each next token from its predicted probabilities, rather than always taking the single most likely one.
Sampling is what makes two answers to the same prompt differ. Settings such as temperature and top-p tune it.
Frequently asked questions
What does Sampling mean?
How a model picks each next token from its predicted probabilities, rather than always taking the single most likely one. Sampling is what makes two answers to the same prompt differ. Settings such as temperature and top-p tune it.