How Do We Know the Flu Shot Works? The Test-Negative Design

Each autumn, flu vaccines roll out, and each winter someone publishes a figure such as "50% effective." Behind that number is a clever design that few people have heard of. It lets researchers estimate protection from routine clinic data, without recruiting thousands of volunteers for a new trial.

Efficacy versus effectiveness

The CDC distinguishes the two. Efficacy comes from randomized trials, while effectiveness describes real-world protection, and observational studies are subject to biases such as confounding that trials largely avoid (CDC). The test-negative design is the standard way many programs estimate effectiveness.

Cases and controls, from the same clinic

Everyone enrolled has a similar respiratory illness and is tested for influenza. Those who test positive are the cases, and those who test negative are the controls. The comparison is the odds of being vaccinated in each group.

Take a hypothetical clinic with 300 positives and 700 negatives. Among positives, 90 were vaccinated (30%), against 350 of the negatives (50%). The odds ratio calculator gives an odds ratio of 0.43. Vaccine effectiveness is one minus the odds ratio, so about 57%. A 95% interval runs from roughly 43% to 68%. The chi-square test for independence returns p < 0.001.

Why choose negative controls?

People who seek care when ill may differ from those who stay home, and vaccinated people may be more likely to seek care. Using patients with the same symptoms but a negative test makes the groups more comparable on care-seeking behavior. That is the design's main appeal.

What can still go wrong

The design rests on assumptions. Vaccination should not change the risk of other respiratory viruses, tests should be accurate, and calendar time should be accounted for, since estimates can be biased if analyses ignore it (Jackson and Rothman, 2015). Small samples widen intervals, which is why the Fisher exact test matters for rare subgroups. Our guide to reading a clinical trial explains the trial side of the story.

Another limitation is that one odds ratio averages over groups. Protection may differ between children and older adults, and waning over months can shift estimates late in the season, which is why programs report results by age group and time since vaccination. Even so, the design remains a pragmatic compromise: faster and cheaper than a trial, and honest about its assumptions when used well.

Vaccine effectiveness also changes year to year, because circulating viruses and vaccine matches change.

A single seasonal percentage is a snapshot, not a constant.

Reading a published estimate

Always read the interval, not just the percentage. One US outpatient study across three seasons enrolled 10,650 patients and reported overall effectiveness of 47%, with a 95% interval from 42% to 52% (CDC Stacks). The large sample gave a narrow interval. Estimates for a small age group or a single week would be far wider.

Outcomes matter too. CDC notes that estimates for non-specific outcomes are generally lower than those for laboratory-confirmed influenza, so two studies can both be correct and still differ. Real analyses also adjust for age, sex, health status and calendar time, which a simple two-by-two table cannot do.

The numbers here are illustrative and not medical advice. Questions about vaccination belong with a doctor or pharmacist.