PeerEval: Ranking Language Models by Peer Review
What if the models being evaluated were also the evaluators? PeerEval has every model answer the same questions, then has every model judge anonymous pairs of answers, and turns all of those verdicts into a leaderboard, weighting each judge by how well its own answers are rated by its peers.
- Every model is both a contestant and a judge. Judges never see who wrote an answer, and every pair is judged in both display orders.
- Verdicts are combined with PRD Peer Rank, which gives more weight to judges whose own answers are rated highly, and are reported with 95% bootstrap confidence intervals and Elo-scale ratings.
- First release: 7 models, 3 benchmarks, 137 questions, 40,156 pairwise judgments. See the leaderboard.
- Judges are biased: every model favors its own answers, by up to 36 percentage points. Peer weighting reduces the influence of the least reliable judges.
- Against human votes, PRD beats single judges and the unweighted panel.
- The method is available as an open-source package:
pip install peereval.
Why peer evaluation?
Open-ended tasks, such as answering a hard user request or writing a story, have no single correct answer, so they are usually evaluated in one of two ways. Human preference, as in Arena, is the gold standard but slow and expensive. An LLM judge is fast and cheap, but a single judge brings its own taste and biases into the ranking. Arena recently reported that frontier judges pick their own answers far more often than humans do. Some leaderboards instead use a small fixed panel of frontier judges; Artificial Analysis's GDPval-AA, for example, uses three.
PeerEval takes the panel idea to its end. Every model being ranked also sits on the panel, so no single company's model decides the result. That raises an obvious question, which is how much to trust each judge. PeerEval's answer is to let the data decide: a judge earns influence through the quality of its own answers, as rated by everyone else.
How it works
1. Answer
Every model answers the same fixed set of questions with the same prompt and no system prompt. Only the final answer is kept: hidden reasoning is never shown to the judges.
2. Judge
For each question, every pair of answers is shown to every model, which picks the better one or calls a tie and returns a short JSON verdict. The judge is told the answers are anonymous and is not told which models wrote them. Each pair is judged twice, once in each display order, so a judge's preference for whichever answer comes first cancels out.
%.inc: %.t FILE … (The recipe line must start with a tab, not spaces.) ….t.inc) is parsed as an old-style suffix rule, and suffix rules can't have prerequisites … Use a pattern rule instead: %.inc: %.t FILE …{"winner": "A", "confidence": 0.64}"Both give the correct pattern rule and explain the automatic variables clearly. A is slightly more complete: it notes the required recipe tab and provides a portable alternative. B's suggestion to delete suffix declarations could break other suffix rules in the Makefile."
3. Rank with PRD Peer Rank
Counting wins treats every judge equally. PRD Peer Rank (Li et al., 2023) instead assumes that a model that gives better answers is also a better judge, and lets each judge's influence follow from its own results:
- All judges start with equal weight.
- A model's score is the weighted average, over judges, of how often each judge picks it.
- Each model is also a judge: its score, scaled to 0–1 and normalized across judges, becomes its new judge weight.
- Steps 2 and 3 repeat until the weights stop changing.
Scoring a model (step 2). A model's score is a weighted average of what each judge thinks of it:
- pj(m): among all comparisons involving model m that judge j decided, the share where j picked m (a tie counts as half). This is judge j's opinion of m.
- wj: the weight of judge j, i.e. how much its opinion counts.
- s(m): the result, between 0 and 1. A judge with twice the weight counts twice as much; with equal weights this is just the average of the judges' opinions.
Updating the judge weights (step 3). Every model is also a judge, so its score as a contestant sets its weight as a judge. The scores are first rescaled so the best model gets 1 and the worst gets 0, and then divided by their total so that all weights add up to 1:
- s̃j: judge j's own score, rescaled to the range 0–1 (the min and max are taken over all models k).
- wj: its share of the total, which becomes judge j's new weight.
With the new weights, the scores are recomputed (step 2), which gives new weights again (step 3), and so on until nothing changes. In practice this takes about ten rounds. In our data, the final weights range from 0.23 for Claude Opus 5.5, the strongest model, down to 0 for Muse Glimmer 30B: because of the 0–1 rescaling, the lowest-scoring model always ends with a judge weight of zero, so its votes no longer affect the ranking.
4. Report uncertainty
- 95% confidence intervals. We resample the questions with replacement 1,000 times (within each benchmark) and recompute every score; the interval covers the middle 95%.
- Rank (UB). The best rank a model could plausibly have: 1 + the number of models that are clearly better, meaning their entire interval lies above this model's. For example, GLM-5.3 ranks second by score, but its gap to Claude Opus 5.5 is within the uncertainty, so its Rank (UB) is 1: the two are statistically tied. Models that share a Rank (UB) cannot be told apart with the current data.
- Elo. A Bradley-Terry model fitted to the same verdicts, shown on the Elo scale: a 100-point gap means the higher model is expected to win about 64% of comparisons.
Benchmarks
Peer review is most useful where there is no answer key, so the leaderboard focuses on open-ended tasks that are normally graded by a judge. The first release uses three benchmarks, 50 questions each (sampled with a fixed seed), covering 137 questions that every model answered validly:
| Benchmark | What it tests | Answer key? | Questions |
|---|---|---|---|
| Arena-Hard v2 | Hard, real user requests: coding, writing, analysis | No | 48 |
| EQ-Bench Creative Writing v3 | Long-form creative writing under constraints | No | 50 |
| Humanity's Last Exam | Expert-level reasoning and knowledge | Yes | 39 |
Humanity's Last Exam is the one benchmark with an answer key. We keep it because it lets us check whether judges can still recognize the correct answer on hard, objective questions.
The leaderboard
Seven models took part in the first release, each as contestant and judge. Overall results across all three benchmarks:
| Rank | Model | PRD score | 95% CI | Rank (UB) | Elo | Unweighted win rate | Judge weight |
|---|---|---|---|---|---|---|---|
| 1 | Claude Opus 5.5 | 0.767 | 0.734–0.794 | 1 | 1123 | 0.742 | 0.231 |
| 2 | GLM-5.3 | 0.719 | 0.692–0.743 | 1 | 1100 | 0.715 | 0.215 |
| 3 | GPT-6 Astra | 0.662 | 0.637–0.687 | 3 | 1040 | 0.637 | 0.197 |
| 4 | Kimi K3 | 0.633 | 0.609–0.656 | 3 | 1026 | 0.619 | 0.187 |
| 5 | Grok 4.6 | 0.356 | 0.331–0.380 | 5 | 836 | 0.367 | 0.096 |
| 6 | Qwen3.8-27B | 0.291 | 0.257–0.325 | 6 | 785 | 0.306 | 0.074 |
| 7 | Muse Glimmer 30B | 0.066 | 0.055–0.078 | 7 | 585 | 0.111 | 0.000 |
- Two clear tiers. Claude Opus 5.5 and GLM-5.3 share Rank (UB) 1, and GPT-6 Astra and Kimi K3 share Rank (UB) 3: within each pair, the gap is inside the uncertainty of 137 questions.
- The method does not drive the ranking. PRD score, unweighted win rate and Elo give the same order. PRD mainly changes how much each judge counts, not who wins.
- Rankings shift by task. Opus 5.5 leads on Arena-Hard v2 and Humanity's Last Exam, while GLM-5.3 and GPT-6 Astra are essentially tied at the top of creative writing. Per-benchmark tables are on the leaderboard.
Meta-evaluation: can peer review be trusted?
A leaderboard built from model verdicts is only worth as much as those verdicts, so we checked them against cases where the answer is known: real human votes.
We took 805 battles from LMArena where a user saw two anonymous answers and voted for one. 420 come from the crowd set and 385 from the expert set. Four judges scored every battle, never shown the vote, each pair judged in both display orders. A judge's accuracy is the share of battles where it picked the answer the human picked. The unweighted panel averages the four verdicts equally. The panels use no human labels, only the judges' verdicts on each other.
Agreement with human votes
PRD leads both sets. Another useful comparison is against the average single judge, which is what you get if you pick one without knowing which is best: PRD also beats it.
What the judges are actually rewarding
The charts below measure each method against the human rate: zero means it shares the humans' preference, and either direction means it over- or under-weights the feature.
PRD stays within 1.3pp of the human rate on every measure and both sets; the
unweighted panel stays within 4.1pp. Only two single-judge bars come closer than PRD
on any measure. GPT-6 Astra is the
outlier throughout, under-weighting length by 21pp on crowd votes and
14pp on expert votes.
Family preference follows the same shape. GPT-6 Astra leans toward Claude answers by
+31pp on crowd votes and +9pp on expert votes; both
panels largely cancel it out, with PRD at +1.1pp and −1.3pp
and the unweighted panel at +3.2pp and −0.6pp.
Recovering the human ranking
What we learned about the judges
Every model favors its own answers
Self-bias compares how often a model prefers its own answer when it is the judge with how often the other judges prefer that same model, in percentage points. Every model is positive:
Arena found the same pattern independently on real user battles, where GPT-6 Astra picked its own answer in 88% of cases. It is the main reason a single model should not grade its own family's models, and one motivation for spreading judgment across the whole panel.
Some judges mostly follow the display order
A reliable judge should give the same verdict when the two answers are swapped, and should pick the first answer about half the time.
| Judge | Same verdict after swapping A and B | First answer wins (decisive verdicts) |
|---|---|---|
| GPT-6 Astra | 93.8% | 49.2% |
| Claude Opus 5.5 | 92.5% | 48.0% |
| Kimi K3 | 88.5% | 50.9% |
| GLM-5.3 | 85.5% | 49.7% |
| Grok 4.6 | 85.0% | 53.7% |
| Qwen3.8-27B | 78.0% | 56.3% |
| Muse Glimmer 30B | 54.3% | 70.4% |
Muse Glimmer 30B picks the first answer 70% of the time and keeps its verdict after a swap only 54% of the time, barely better than chance. Judging both orders keeps this from distorting the comparisons, and PRD gives Muse a judge weight of zero, so its votes do not affect the PRD scores at all.
Peer weighting concentrates trust in the strongest judges
The four strongest models end up with 83% of the total judge weight, spread fairly evenly (0.19–0.23 each), so no single company's judge dominates. The three weakest judges together hold 17%. In this release the weighted and unweighted rankings agree; the weighting matters most when weak judges disagree with strong ones, which the meta-evaluation above examines.
Use PeerEval yourself
The scoring and judging code is open source as the peereval Python package. If you already have pairwise judgments, scoring needs no API keys:
pip install peereval
peereval score judgments/*.jsonl --bootstrap 1000 --json results.json
To produce judgments from your own models' answers through any OpenAI-compatible endpoint (OpenAI, OpenRouter, vLLM, SGLang):
pip install "peereval[judge]"
peereval generate questions.jsonl --models models.json --out answers.jsonl
peereval judge answers.jsonl --judges models.json --out judgments.jsonl
Code and documentation: github.com/peereval-ai/peer_rank_tool · Package: pypi.org/project/peereval