95% is the industry standard. Use 99% when the cost of a wrong decision is high (e.g., pricing changes). Use 90% only for fast, low-risk iterations.
What Is A/B Testing?
A/B testing (also called split testing) is a controlled experiment that compares two versions of something — a webpage, email, button, or any other element — to determine which performs better on a chosen metric. One group of users sees Variant A (the control), another sees Variant B (the challenger). The key question is: Is the observed difference in conversion rates real, or could it be random noise?
Statistics gives us a principled answer through hypothesis testing.
The Null and Alternative Hypotheses
Before running the test, we define two competing claims:
- H₀ (Null Hypothesis): There is no real difference between A and B. Any observed difference is due to chance.
- H₁ (Alternative Hypothesis): There is a real difference — B's conversion rate differs from A's.
We never "prove" H₁; we only gather evidence strong enough to reject H₀.
Conversion Rate
For each variant, the observed conversion rate is simply:
Example: 210 conversions from 5,000 visitors → p̂ = 0.042 = 4.2%
The Z-Test for Two Proportions
We use a two-proportion Z-test to measure how many standard deviations the observed difference lies from zero (i.e., from "no effect").
Step 1 — Pooled proportion (the best estimate of the true rate under H₀):
Step 2 — Standard error of the difference:
Step 3 — Z-score:
A large |Z| means the difference is far from zero in standard-deviation units — strong evidence against H₀.
The p-value
The p-value is the probability of observing a difference at least as extreme as the one measured, assuming H₀ is true. We use a two-tailed test because we care about differences in either direction:
where \(\Phi\) is the standard normal cumulative distribution function.
- If p < α (e.g., p < 0.05 for 95% confidence), reject H₀ → result is statistically significant.
- If p ≥ α, we do not have enough evidence to reject H₀.
Confidence Level
The confidence level (1 − p) tells you how certain we are that the observed difference is not random:
A 95% confidence level means: if we ran this exact experiment 100 times, 95 of those experiments would produce a significant result if the true effect is real.
Relative Uplift
If B outperforms A, the relative uplift quantifies the improvement:
This is more meaningful than absolute difference — a 1% → 1.1% improvement (+0.1 pp) is actually a +10% relative uplift.
Worked Example
| Metric | Variant A (Control) | Variant B (Challenger) |
|---|---|---|
| Visitors (N) | 5,000 | 5,000 |
| Conversions (C) | 210 | 265 |
| Conversion Rate (p̂) | 4.20% | 5.30% |
Pooled proportion:
Standard error:
Z-score:
p-value:
Since p = 0.0096 < 0.05, the result is statistically significant at the 95% confidence level. Variant B's higher conversion rate is unlikely to be due to chance. Relative uplift: +26.2%.
Common Pitfalls
- Peeking early: Checking results before your planned sample size is reached inflates false positives. Commit to a minimum sample before stopping.
- Small samples: The Z-test approximation is reliable when both N·p̂ ≥ 5 and N·(1−p̂) ≥ 5 for each variant.
- Multiple testing: Running many simultaneous tests increases the chance of a false positive. Apply a Bonferroni correction if needed.
- Novelty effect: Users may respond differently to something simply because it's new. Allow the test to run long enough to capture stable behavior.
- Significance ≠ importance: A statistically significant result can still have a negligible business impact. Always pair the p-value with practical effect size.