Can You Trust the Polls? What the 2020 Polling Error Showed

Every election cycle, the same argument returns: the polls were wrong, so why trust them now? With midterm elections on 3 November, it is worth asking what "wrong" actually means. The 2020 polling error is the best-documented case, and its lessons are less dramatic, and more useful, than the slogans on either side.

What the 2020 review found

A task force of the American Association for Public Opinion Research (AAPOR) examined thousands of 2020 polls and described the error as "of an unusual magnitude": the highest in 40 years for the national popular vote and in at least 20 years for state-level polls (AAPOR). Averaged over the final two weeks, national polls overstated the Democratic candidate's lead by about 3.9 points, and state polls by about 4.3. The polls did point to the correct national winner.

Margin versus share

A miss sounds large partly because polls report a margin, the gap between two candidates, and the error on a margin is about double the error on either share. A poll showing 52% to 46%, a six-point lead, against a result of 50% to 48% is off by two points on each candidate but four on the margin. The difference of two proportions calculator and the two-proportion confidence interval calculator show how wide the uncertainty on a gap really is.

Why a shared error is the real problem

Margins of error describe random sampling noise, which tends to average out across polls. Shared bias does not average out. Suppose ten polls each have independent errors of about 2.5 points. The chance that eight or more miss in the same direction is about 11%, the same as for a fair coin in the binomial distribution calculator.

Add a common error of about 3 points that affects every poll, and that chance jumps to roughly 60%. Many polls missing together is what shared bias looks like.

Averaging polls reduces noise. It cannot remove a bias that all of them share.

Nonresponse, not shy voters

The task force could not name a single culprit, but it ruled out several popular explanations, including late deciders and voters who hid their preferences. It concluded that at least some of the error came from nonresponse, meaning systematic differences between people who answer polls and people who do not (Pew Research Center). Pollsters had largely fixed one problem from 2016, the underrepresentation of voters without college degrees, yet the error persisted.

How to read a poll now

Treat a poll as a measurement with two kinds of error: sampling noise, which the confidence interval for a proportion can estimate, and bias, which nobody can see until the votes are counted. A lead of one or two points is within reach of both.

Our article on election forecast probabilities explains how forecasters turn polls into odds, and our piece on poll margins of error covers the sampling side. The AAPOR review itself noted that most polls lack the precision needed to call semi-close races.

So can you trust polls? Trust them for what they measure: the rough shape of public opinion, with an uncertainty larger than the printed margin. Distrust any claim that treats a two-point lead as a certainty, or any claim that polls are worthless. Both are statistical overreach.