The Replication Crisis: Why Some Published Findings Don't Repeat

A study finds an effect, gets published and makes headlines. Then another team repeats it and finds nothing. This has happened often enough that researchers gave it a name: the replication crisis. Much of the explanation is statistical, and you can check it yourself with a few numbers.

What the large replication projects found

In 2015 the Open Science Collaboration tried to replicate 100 psychology studies. It reported that the replication effects averaged about half the size of the original effects, and that far fewer replications reached statistical significance than the originals had. Earlier, John Ioannidis argued that when studies are small, effects are modest, many relationships are tested and analyses are flexible, a significant published result can be more likely false than true. Neither paper says science is broken. Both show that a single significant result is weaker evidence than it looks.

Small samples are noisy

Suppose a study of 10 people finds a correlation of r = 0.5. The p-value is about 0.14, and the 95% confidence interval for the true correlation runs from about −0.19 to +0.86: anything from slightly negative to very strong. With 30 people the same r = 0.5 gives p ≈ 0.005. Experiment with sample size in the correlation coefficient simulator and see how wildly r jumps when n is small, then test your own data with the Pearson correlation calculator. Because journals tend to favor significant results, the estimates that get published are often the exaggerated ones.

Many tests, one headline

Test 20 unrelated outcomes at the 5% level when nothing is going on, and the chance of at least one "significant" result is 1 − 0.95²⁰ ≈ 64%. With 10 tests it is about 40%. A Bonferroni correction divides the threshold by the number of tests: 0.05 ÷ 20 = 0.0025. If only the one lucky hit is reported, readers cannot tell it came from a lottery.

One significant result is a question. A replicated one is an answer.

What a p-value is, again

A p-value is the probability of data at least as extreme as those observed, assuming the null hypothesis and the model assumptions hold. The American Statistical Association's statement stresses that p-values do not measure the probability that a hypothesis is true or the size of an effect, and that conclusions should not rest only on whether a p-value crosses a threshold. For a plain-language walkthrough, see What Is a P-Value?

Habits that make results repeatable

Plan the sample size before collecting data (our sample size guide shows how). Decide in advance which test is the main one, and report how many tests you ran. Give effect sizes with confidence intervals, for example with the confidence interval calculator for a mean, and compare groups with the independent samples t-test calculator. Treat a single surprising study as a hypothesis to be replicated, not a settled fact.