Can we automate
policy evaluation?

The Tournament

Papers compete in head-to-head matches. Each day at 9:00 AM UTC, we run a round of matches. An LLM judge reads both papers and picks one as preferred (or declares a tie). The preferred paper gains rating points; the other loses points. Over time, papers consistently favored by the LLM judge rise in the rankings — though this is a noisy and potentially biased signal, not ground truth.

How Matches Work

For each match, we send both paper PDFs directly to Gemini 3.1 Flash Lite (Google's LLM). The model sees the full papers — text, figures, tables, formatting — exactly as a human reviewer would.

The judge is prompted to act as a "senior editor at a top economics journal" and evaluate papers on: identification strategy (is the causal inference credible?), novelty, policy relevance, execution quality, and appropriate scope. View the full prompt.

To control for position bias (LLMs sometimes prefer whichever paper they see first), we run each comparison twice with the papers swapped. A paper must win both rounds to win the match; otherwise it's a tie.

Rating System

Rankings use TrueSkill, a Bayesian rating system developed by Microsoft. Each paper has two numbers:

• μ (mu) — estimated skill level. Higher = paper wins more often.
• σ (sigma) — uncertainty. Decreases as the paper plays more matches.

Papers are ranked by their conservative rating (μ − 3σ), which represents the lower bound of estimated skill. This means papers need consistent wins across multiple matches — a single lucky win won't send a paper to the top.

Head-to-Head Statistics

Win counts exclude 896 ties where the judge couldn't determine a clear winner.

Prob(Human Win) answers: if we randomly pick one human paper and one APE paper, what's the probability the human paper wins according to the LLM judge? Computed from TrueSkill ratings, accounting for uncertainty — papers with fewer matches contribute less certainty. The main metric compares recent cohorts (last 25 of each); all-time (89.4%) includes all 909 AI and 43 human papers with 5+ matches.

Matchup Selection

Each day we run 50 matches (100 LLM calls with position swapping) in 10 batches of 5. Within each batch, no paper plays twice. We combine random matching with structured matching.

Important Caveats ⚠️

The ⚠️ warning icon in the leaderboard indicates AI-generated papers that have not been peer reviewed. The LLM judge is not a substitute for human peer review. AI-generated papers may contain errors, hallucinations, or fabricated results. In fact, we have found that these are very common and sometimes take a lot of effort to spot. If a paper looks too good to be true, it probably is.

The judge evaluates the PDF only, not the underlying code or data. Rankings should not be taken at face value. That's why everything is open source — code, data, and papers are all public so anyone can spot errors, report issues, and contribute improvements.