Is Model A Really Better Than Model B? Testing AI Benchmark Scores

Every few weeks a new AI model tops a leaderboard by a point or two, and a press release calls it a breakthrough. Statisticians ask a simpler question: could the gap just be luck?

A benchmark score is a sample

A model scoring 82.0% on a 1,000-question test is not an "82% model." It is an estimate from one sample of questions. The confidence interval for a single proportion puts that score between roughly 79.6% and 84.4%. A rival at 80.5% sits comfortably inside that range.

Treat the two models as independent groups and the difference of two proportions calculator gives p ≈ 0.39. By that measure, there is no evidence of a real difference.

Question difficulty matters too. If most questions are easy for both models, the few that separate them carry nearly all the information, which is why a headline gap can hide a very small amount of real evidence. Benchmark size, not just accuracy, decides how much a score can be trusted.

Same questions, smarter test

Both models answered the same questions, so the data is paired, and the informative cases are those where they disagree. McNemar's test uses exactly those. Suppose A beats B on 90 questions and B beats A on 75, which matches the 1.5-point gap. The p-value is about 0.28, still inconclusive, yet this paired approach usually has more power than comparing two overall percentages.

Many models, many runs

Leaderboards compare dozens of systems, and each model's score varies from run to run. To compare three or more groups of scores, use the one-way ANOVA calculator, or Kruskal-Wallis when scores are far from normal. For the same tasks scored under two settings, the Wilcoxon signed-rank test is a natural fit.

Remember multiple comparisons too. With enough models on a board, someone will come out on top by chance alone, and that winner is rarely the best model in any lasting sense.

Finally, treat contamination and test reuse as statistical issues as well. If models are tuned repeatedly against the same public test set, the scores gradually stop being honest estimates of performance on new questions, however large the benchmark looks.

Putting error bars on the leaderboard

The simplest improvement is to report intervals. With 1,000 questions, an accuracy near 82% carries a margin of roughly ±2.4 points. Increase the benchmark to 10,000 questions and the margin shrinks to about ±0.75, since precision improves with the square root of the sample size, not in proportion to it.

Practical importance deserves its own question. A one-point gain is trivial for a casual chatbot, but it can matter when a system handles millions of requests. Significance tells you whether a difference is likely real, while effect size tells you whether anyone should care.

The details that change the score

Report what makes a score comparable: the exact prompt format, the number of attempts, the sampling temperature and the scoring rule. Two labs can run the same benchmark and obtain different numbers from these choices alone, so a gap between published scores may reflect procedure rather than ability.

When possible, rerun each model several times and report the spread, since sampling randomness alone can move a score by a fraction of a point. Better still, publish per-question results so others can run paired tests on the exact same items.

A better question than "who is first?"

Ask instead: by how much, measured on how many questions, and with what uncertainty?

A one-point lead on a small benchmark is a conversation starter, not a conclusion.