Evaluation
LLM as a judge
Using a language model to grade other models' outputs against criteria, to evaluate at scale.
Model judges are fast and cheap, but have biases, for example toward longer answers or their own style. Calibrate them against some human-graded examples.
Frequently asked questions
What does LLM as a judge mean?
Using a language model to grade other models' outputs against criteria, to evaluate at scale. Model judges are fast and cheap, but have biases, for example toward longer answers or their own style. Calibrate them against some human-graded examples.