How Fair Is Gymnastics Judging? Measuring Agreement Between Judges

When the world's best gymnasts compete in Rotterdam from 17 to 25 October, a panel of judges will score every routine. Fans often argue that a mark felt too high or too low. Statistics cannot read a judge's mind, but it can measure how closely judges agree, and whether their disagreement looks like noise or bias.

Two judges, one ranking

Imagine two judges ranking eight gymnasts. Judge A's order is 1 to 8, while Judge B swaps three neighboring pairs. The Spearman correlation calculator returns about 0.93, and the Kendall tau calculator gives 0.79. Kendall's value is lower because it counts every pair the judges order differently, here 3 of 28. Both signal strong agreement, but they punish small swaps differently.

Rank agreement has a blind spot, though. Two judges can rank everyone identically while one scores half a point lower throughout. Ranks hide generosity, and scores reveal it.

Do judges differ in leniency?

To test generosity, compare the same eight routines across three judges. The Friedman test calculator checks whether judges differ systematically while treating each gymnast as a block. In an illustrative panel where one judge is consistently 0.3 points harsher, it returns p ≈ 0.002, flagging the harsh judge even though all three agree on who is best.

For a single routine, the standard deviation calculator shows the spread. Marks of 8.6, 8.8, 8.7, 9.2 and 8.5 average 8.76 with a standard deviation of 0.27, and the 9.2 stands out. Some scoring systems drop the highest and lowest marks before averaging, a simple defence against one extreme judge, at the cost of a little information.

Reliability, bias and home crowds

Researchers often summarize a panel with the intraclass correlation, which asks what share of score variation reflects real differences between gymnasts rather than judge noise (Shrout and Fleiss, 1979). Values near 1 mean judges are nearly interchangeable.

Bias is a different question. An analysis of Summer Games from 1896 to 1996 found significant home advantage in subjectively judged events such as gymnastics and boxing, but little in objectively measured ones like athletics (Balmer, Nevill and Williams, 2003). Our piece on Olympic home advantage shows how such effects are tested.

Agreement tells you judges are consistent. It does not tell you they are right.

Small panels, big uncertainty

A panel average carries its own error. With five judges and a standard deviation of 0.27 between them, the average of the panel is uncertain by about 0.12 points (the standard deviation divided by the square root of five), or roughly ±0.24 at the usual 95% level. A gap of 0.1 between two gymnasts is therefore smaller than the judging noise attached to each average.

That does not make results meaningless. It means narrow margins deserve humility, and that decisions made on tenths of a point are partly decisions made by chance. Larger panels, clearer criteria and video review all shrink that noise, and statistics lets us measure by how much.

So when a controversial score appears in Rotterdam, ask how large the gap is, how large the usual spread between judges is, and whether one judge differs systematically across many routines. One surprising score is a story. A pattern across a hundred is evidence. Publishing the spread of the judges' marks would help fans judge the judges.