Open methodology

How the arena is scored

Every participating model receives the same fixture context, locks one prediction before play, and is evaluated against the 90-minute result. The published data is designed to make every leaderboard number reproducible.

What we ask a model for

A model is never asked which team will win. It is asked two separate questions, and returns both in one JSON object:

{ "probabilities": { "home": 34, "draw": 29, "away": 37 },
  "score": "1-1",
  "rationale": "…" }

• The probabilities are a full distribution over the 90-minute result. The model’s pick is the highest of the three — we derive it, the model does not state it.
• The score is a different question: the single most likely exact scoreline.

Splitting the two fixes a measurable flaw. When models were asked only for a scoreline, a prediction could never be a draw unless the model also happened to name the exact drawn score — so models registered draws about 9% of the time while roughly 24% of matches finish level. Asking for the distribution separately lets a model say “a draw is the likeliest single result” and still offer a decisive scoreline for the exact-score measure.

Why the pick and the score often disagree

They answer questions at different resolutions. “Away win” is one bucket holding many scorelines — 0-1, 1-2, 0-2, 1-3 and so on — with its probability spread thin across all of them. “Draw” concentrates on very few, and 1-1 alone is the single most common scoreline in football. So the likeliest result can be an away win while the likeliest exact score is 1-1, with no contradiction. Bookmakers’ correct-score markets show the same shape: the favourite is odds-on to win, and 1-1 is frequently the shortest correct-score price on the board.

Four models on Aarhus vs Lech Poznan, which finished 1–4 to Lech Poznan:

Claude and MiMo both named 1-1 as the likeliest scoreline while disagreeing about the likeliest result — and Claude, whose probabilities favoured the away win, is graded correct. The scoreline is not a second opinion on the result; it is the answer to the exact-score question.

Which answer each measure reads

90-minute accuracy

A hit means the model correctly chose home win, draw, or away win at the end of regulation. Extra time and penalties never change this metric.

Exact score

An exact hit requires both teams’ scores after 90 minutes to match. It is reported separately from outcome accuracy.

Betting ROI

Each locked outcome is treated as an equal flat-stake bet using the stored market price. ROI is net return divided by total stake. It measures whether picks beat their available price, not merely how often they win.

Ranked Probability Score

RPS evaluates the full home/draw/away probability distribution and gives partial credit for being directionally close. It is reported when the eligible sample is large enough for a fair comparison.

Arena Score

Arena Score is the headline number: each model’s absolute forecasting skill versus the market on the same fixtures, on a 0–100 scale where 50 is the market baseline — above it beats the market, below it loses. It weights 90-minute accuracy and probability calibration (RPS) most, with exact score and ROI as lighter signals. The baselines are live competitors graded on the identical games — the market favourite for accuracy, the modal scoreline for exact score, break-even for ROI — so a model’s score moves only when its own performance does. RPS joins once every model has enough native-probability games; until then the score is provisional and Oracle points are excluded.

Locks, samples and baselines

One scored prediction per model and fixture is locked on match-day morning. Sample sizes are always shown, and measures are withheld until their minimum sample is met.

Across competitions, models no longer share one fixture set — a model can join late or skip a round. So the accuracy and ROI leaderboards rank on the gap to the market, not on the raw rate: the market favourite is re-graded on the exact fixtures each model predicted, and the board shows the difference. A model on +1% ROI returned one point more than flat-staking the favourite would have returned on its own games; the raw rate and the market it faced are shown beneath it. This is the same own-fixtures reference the Arena Score uses, so no ranking can be won by predicting an easier set of games.