PeerEval: Ranking Language Models by Peer Review

What if the models being evaluated were also the evaluators? PeerEval has every model answer the same questions, then has every model judge anonymous pairs of answers, and turns all of those verdicts into a leaderboard, weighting each judge by how well its own answers are rated by its peers.

In short
Opus 5.5 0.231 GLM-5.3 0.215 GPT-6 Astra 0.197 Kimi K3 0.187 Grok 4.6 0.096 Qwen 27B 0.074 Muse 30B 0.000 Opus 5.5 — judge weight 0.231 GLM-5.3 — judge weight 0.215 GPT-6 Astra — judge weight 0.197 Kimi K3 — judge weight 0.187 Grok 4.6 — judge weight 0.096 Qwen 27B — judge weight 0.074 Muse 30B — judge weight 0.000; PRD discards its votes Judge weight Ring thickness is how much a judge's verdicts count, set by the model's own score. One pairwise battle Two answers to the same question, scored by all seven judges in both orders. Contestant and judge Every model plays both roles, including on its own battles. Hover a model for its battles.
The first release: seven models, each answering every question and judging every pair of answers, including pairs its own answer is in. Judge weights are the values PRD converges on after about ten rounds; Muse Glimmer 30B lands at zero, so its verdicts drop out of the ranking entirely.

Why peer evaluation?

Open-ended tasks, such as answering a hard user request or writing a story, have no single correct answer, so they are usually evaluated in one of two ways. Human preference, as in Arena, is the gold standard but slow and expensive. An LLM judge is fast and cheap, but a single judge brings its own taste and biases into the ranking. Arena recently reported that frontier judges pick their own answers far more often than humans do. Some leaderboards instead use a small fixed panel of frontier judges; Artificial Analysis's GDPval-AA, for example, uses three.

PeerEval takes the panel idea to its end. Every model being ranked also sits on the panel, so no single company's model decides the result. That raises an obvious question, which is how much to trust each judge. PeerEval's answer is to let the data decide: a judge earns influence through the quality of its own answers, as rated by everyone else.

How it works

1 Answer 2 Judge 3 Rank 4 Report Every model answers every question Every model compares every anonymous pair, A/B and B/A PRD Peer Rank weights the judges Scores, 95% CI, Elo, Rank (UB) repeat until weights converge

1. Answer

Every model answers the same fixed set of questions with the same prompt and no system prompt. Only the final answer is kept: hidden reasoning is never shown to the judges.

2. Judge

For each question, every pair of answers is shown to every model, which picks the better one or calls a tie and returns a short JSON verdict. The judge is told the answers are anonymous and is not told which models wrote them. Each pair is judged twice, once in each display order, so a judge's preference for whichever answer comes first cancels out.

Question (Arena-Hard v2)
GNU make says "warning: ignoring prerequisites on suffix rule definition" for my Makefile. How do I change it so that I don't get that warning but I can still say that files with some suffix .t have a dependency on a given file called FILE? …
Response A (anonymous)
Replace the suffix rule with a GNU make pattern rule. Pattern rules can have extra prerequisites: %.inc: %.t FILE … (The recipe line must start with a tab, not spaces.) …
Response B (anonymous)
The warning happens because a target with two suffixes (.t.inc) is parsed as an old-style suffix rule, and suffix rules can't have prerequisites … Use a pattern rule instead: %.inc: %.t FILE …
Verdict
{"winner": "A", "confidence": 0.64}
"Both give the correct pattern rule and explain the automatic variables clearly. A is slightly more complete: it notes the required recipe tab and provides a portable alternative. B's suggestion to delete suffix declarations could break other suffix rules in the Makefile."
A real verdict from the first release. Behind the scenes: A was Claude Opus 5.5, B was GLM-5.3, and the judge was GPT-6 Astra; none of this was visible to the judge.

3. Rank with PRD Peer Rank

Counting wins treats every judge equally. PRD Peer Rank (Li et al., 2023) instead assumes that a model that gives better answers is also a better judge, and lets each judge's influence follow from its own results:

  1. All judges start with equal weight.
  2. A model's score is the weighted average, over judges, of how often each judge picks it.
  3. Each model is also a judge: its score, scaled to 0–1 and normalized across judges, becomes its new judge weight.
  4. Steps 2 and 3 repeat until the weights stop changing.

Scoring a model (step 2). A model's score is a weighted average of what each judge thinks of it:

s(m)= ∑jwjpj(m) ∑jwj

Updating the judge weights (step 3). Every model is also a judge, so its score as a contestant sets its weight as a judge. The scores are first rescaled so the best model gets 1 and the worst gets 0, and then divided by their total so that all weights add up to 1:

s~j= s(j)−minks(k) maxks(k)−minks(k) , wj= s~j ∑ks~k

With the new weights, the scores are recomputed (step 2), which gives new weights again (step 3), and so on until nothing changes. In practice this takes about ten rounds. In our data, the final weights range from 0.23 for Claude Opus 5.5, the strongest model, down to 0 for Muse Glimmer 30B: because of the 0–1 rescaling, the lowest-scoring model always ends with a judge weight of zero, so its votes no longer affect the ranking.

4. Report uncertainty

Benchmarks

Peer review is most useful where there is no answer key, so the leaderboard focuses on open-ended tasks that are normally graded by a judge. The first release uses three benchmarks, 50 questions each (sampled with a fixed seed), covering 137 questions that every model answered validly:

BenchmarkWhat it testsAnswer key?Questions
Arena-Hard v2Hard, real user requests: coding, writing, analysisNo48
EQ-Bench Creative Writing v3Long-form creative writing under constraintsNo50
Humanity's Last ExamExpert-level reasoning and knowledgeYes39

Humanity's Last Exam is the one benchmark with an answer key. We keep it because it lets us check whether judges can still recognize the correct answer on hard, objective questions.

The leaderboard

Seven models took part in the first release, each as contestant and judge. Overall results across all three benchmarks:

RankModelPRD score95% CIRank (UB)EloUnweighted win rateJudge weight
1Claude Opus 5.50.7670.734–0.794111230.7420.231
2GLM-5.30.7190.692–0.743111000.7150.215
3GPT-6 Astra0.6620.637–0.687310400.6370.197
4Kimi K30.6330.609–0.656310260.6190.187
5Grok 4.60.3560.331–0.38058360.3670.096
6Qwen3.8-27B0.2910.257–0.32567850.3060.074
7Muse Glimmer 30B0.0660.055–0.07875850.1110.000

Meta-evaluation: can peer review be trusted?

A leaderboard built from model verdicts is only worth as much as those verdicts, so we checked them against cases where the answer is known: real human votes.

We took 805 battles from LMArena where a user saw two anonymous answers and voted for one. 420 come from the crowd set and 385 from the expert set. Four judges scored every battle, never shown the vote, each pair judged in both display orders. A judge's accuracy is the share of battles where it picked the answer the human picked. The unweighted panel averages the four verdicts equally. The panels use no human labels, only the judges' verdicts on each other.

Agreement with human votes

PRD leads both sets. Another useful comparison is against the average single judge, which is what you get if you pick one without knowing which is best: PRD also beats it.

What the judges are actually rewarding

The charts below measure each method against the human rate: zero means it shares the humans' preference, and either direction means it over- or under-weights the feature.

PRD stays within 1.3pp of the human rate on every measure and both sets; the unweighted panel stays within 4.1pp. Only two single-judge bars come closer than PRD on any measure. GPT-6 Astra is the outlier throughout, under-weighting length by 21pp on crowd votes and 14pp on expert votes.

Family preference follows the same shape. GPT-6 Astra leans toward Claude answers by +31pp on crowd votes and +9pp on expert votes; both panels largely cancel it out, with PRD at +1.1pp and −1.3pp and the unweighted panel at +3.2pp and −0.6pp.

Recovering the human ranking

What we learned about the judges

Every model favors its own answers

Self-bias compares how often a model prefers its own answer when it is the judge with how often the other judges prefer that same model, in percentage points. Every model is positive:

Self-bias in percentage points. As a judge, GPT-6 Astra gives its own answers a 94% win rate; the other judges give the same answers 59%.

Arena found the same pattern independently on real user battles, where GPT-6 Astra picked its own answer in 88% of cases. It is the main reason a single model should not grade its own family's models, and one motivation for spreading judgment across the whole panel.

Some judges mostly follow the display order

A reliable judge should give the same verdict when the two answers are swapped, and should pick the first answer about half the time.

JudgeSame verdict after swapping A and BFirst answer wins (decisive verdicts)
GPT-6 Astra93.8%49.2%
Claude Opus 5.592.5%48.0%
Kimi K388.5%50.9%
GLM-5.385.5%49.7%
Grok 4.685.0%53.7%
Qwen3.8-27B78.0%56.3%
Muse Glimmer 30B54.3%70.4%

Muse Glimmer 30B picks the first answer 70% of the time and keeps its verdict after a swap only 54% of the time, barely better than chance. Judging both orders keeps this from distorting the comparisons, and PRD gives Muse a judge weight of zero, so its votes do not affect the PRD scores at all.

Peer weighting concentrates trust in the strongest judges

The four strongest models end up with 83% of the total judge weight, spread fairly evenly (0.19–0.23 each), so no single company's judge dominates. The three weakest judges together hold 17%. In this release the weighted and unweighted rankings agree; the weighting matters most when weak judges disagree with strong ones, which the meta-evaluation above examines.

Use PeerEval yourself

The scoring and judging code is open source as the peereval Python package. If you already have pairwise judgments, scoring needs no API keys:

pip install peereval
peereval score judgments/*.jsonl --bootstrap 1000 --json results.json

To produce judgments from your own models' answers through any OpenAI-compatible endpoint (OpenAI, OpenRouter, vLLM, SGLang):

pip install "peereval[judge]"
peereval generate questions.jsonl --models models.json --out answers.jsonl
peereval judge answers.jsonl --judges models.json --out judgments.jsonl

Code and documentation: github.com/peereval-ai/peer_rank_tool · Package: pypi.org/project/peereval