LLM-as-Judge
Use a capable frontier model to score another model's outputs at scale — assigning quality ratings that would otherwise require human annotators.
Intent & Description
🎯 Intent
Scale quality evaluation beyond what human annotation throughput allows, using a frontier model as a proxy for human judgment on the same outputs.
📋 Context
Human evaluation is slow, expensive, and doesn’t scale to continuous integration pipelines. Automated metrics like ROUGE and BLEU miss quality dimensions like helpfulness, tone, and reasoning quality. An LLM judge bridges the gap — faster than humans, richer than n-gram overlap.
💡 Solution
Define an evaluation rubric covering quality criteria (accuracy, helpfulness, conciseness, safety). Prompt a capable judge model (GPT-4, Claude 3 Opus) with the rubric, the original prompt, and the model’s response. Request a score (1-5 or pass/fail) with a brief rationale. For comparative evaluation, use pairwise preference: show the judge two responses and ask which is better. Calibrate against human annotation on a known subset before relying on the judge.
Real-world Use Case
📌 TL;DR
Use a frontier model as your eval pipeline when human annotation can’t scale. Calibrate against human labels first — LLM judges have known systematic biases you need to measure before trusting.
Advantages
- Scales to thousands of evaluations per hour — infeasible with human annotators
- Captures nuanced quality dimensions (reasoning, tone, helpfulness) that automated metrics miss
- Pairwise comparison format produces reliable relative rankings
Disadvantages
- Judge model has systematic biases — positional bias (favors first response), verbosity bias, self-preference
- Circular evaluation — using one model to evaluate another doesn’t catch their shared failure modes
- Judge quality degrades on tasks outside its own capability ceiling