How to evaluate millions of LLM responses without manual review.
The Problem
Traditional metrics (BLEU, ROUGE) are designed for fixed outputs and don't work for open-ended generation. Manual human review is too slow and costly ($50+/review, days-long delays). You need something that scales.
Core Solution: LLM as a Judge
Use a strong LLM (GPT-4-class or equivalent) to automatically evaluate outputs.
Results: ~85% alignment with human reviewers — better than human inter-rater agreement (~81%).
Evaluation Methods
| Method | What it does | Best for |
|---|---|---|
| Single-output scoring | Rate one response on relevance, accuracy, helpfulness | Ongoing quality monitoring |
| Reference-based | Compare against ground-truth answers | Factual tasks with known correct answers |
| Pairwise comparison | Choose the better of two responses | A/B testing, model comparison |
Enhance with Chain-of-Thought
Make the judge reason step-by-step before scoring:
- Applies criteria systematically
- Outputs scores with justifications
- Uses few-shot examples for consistency
- Reduces position bias and arbitrary scoring
Human-Alignment Metrics (benchmark numbers)
| Dimension | Alignment with human reviewers |
|---|---|
| Factual correctness | 85% |
| Creative quality | 78% |
| Format compliance | 92% |
Recommended Implementations
| Tool/Technique | Use case |
|---|---|
| G-Eval | Custom criteria with step-by-step scoring |
| Pairwise | A/B testing between model versions |
| DAG-based trees | Complex, multi-dimensional evaluations |
| Position swapping | Run pairwise twice with order flipped, average scores |
| Multi-judge consensus | N independent judges, majority vote |
Common Pitfalls
| Bias | Description | Mitigation |
|---|---|---|
| Position bias | Favors the first option in pairwise | Position swapping |
| Verbosity bias | Favors longer responses regardless of quality | Explicit length-neutral criteria |
| Self-preference | Model favors its own output style | Use a different model as judge |
| Temperature sensitivity | Scores vary across runs | Score distributions + probability weighting |
Stand-out Approach (non-deterministic scoring)
Don't just set temperature=0 and call it done. Instead:
- Consensus from multiple independent judges
- Score distributions instead of point estimates
- Probability weighting across judge outputs
- Human baselines — periodically validate judge alignment against fresh human ratings
This is what separates a working eval system from a production-grade one.