What Is a P-Value? What It Means and What It Doesn't

"The result is significant, p = 0.03." Few sentences in science are repeated as often or misunderstood as widely. The p-value is the most famous number in statistics, and the American Statistical Association felt it necessary to issue a formal statement on its proper use (Wasserstein and Lazar, 2016). Here is what the number says, and what it does not.

The correct definition

The p-value is the probability, assuming that a specified statistical model, usually the null hypothesis, is true, of getting data at least as extreme as what you observed. It is a statement about the data in relation to an assumption, not a statement about whether the assumption is true.

A coin-flip example

Suppose you flip a coin 100 times and get 60 heads. If the coin is fair, how surprising is that? The binomial distribution calculator gives a two-sided p-value of about 0.057, a little too large to be called significant at the usual 5% level. With 62 heads, p falls to about 0.021. A difference of two flips moves the result across the line, which shows how arbitrary the threshold is.

A p-value of 0.049 and one of 0.051 carry almost identical evidence. Treating them as "significant" and "not significant" is a convention, not a discovery.

Three common misunderstandings

First, the p-value is not the probability that the null hypothesis is true. Suppose 1,000 hypotheses are tested, 100 of them true effects, with 80% power and a 5% significance level. About 80 true effects are detected, and 45 false positives arise from the 900 null cases. Among the 125 significant results, only 64% are real, despite every p being below 0.05.

Second, a small p-value does not mean a large effect. Imagine two groups of 10,000 whose means differ by 0.5 points with a standard deviation of 15. The p-value is about 0.018, yet the difference is tiny in practical terms. The independent samples t-test calculator lets you try such numbers.

Third, a large p-value does not prove there is no effect. It may only mean the sample was too small to detect one.

How p-values mislead in practice

Multiple testing is a quiet trap. If you run 20 independent tests at the 5% level when nothing is going on, the chance of at least one "significant" result is about 64%. Greenland and colleagues stress that selecting analyses for presentation based on their p-values can yield small p-values even when the tested hypothesis is correct (Greenland et al., 2016).

The ASA statement notes that scientific conclusions should not rest solely on whether a p-value crosses a threshold, and that proper inference needs full reporting and transparency.

A p-value was never meant to replace scientific reasoning.

How to report it

Report the effect size, a confidence interval and the sample size beside the p-value. The one-sample proportion test and the two-proportion confidence interval calculator show how p-values and intervals relate.

Think of a p-value as a measure of how surprising the data would be under one specific assumption. It is useful, limited and easily over-read. Our guide on reading clinical trials shows how to place it in context.