Sample Size Calculations: The Assumptions Nobody Questions
Every sample size calculation is built on assumptions about effect size, variance, and dropout - and most of those assumptions are optimistic. Underpowered trials are not bad luck. They are a predictable consequence of how power calculations are done.
In 2013, Lachenbruch and Slaughter published an analysis of phase III clinical trials in the oncology literature and found that the median time to primary endpoint in the trials was substantially longer than the assumptions used to power them. This was not a new finding - similar analyses had been published before, in cardiology, in psychiatry, in rheumatology. The pattern is consistent: sample size calculations in clinical trials are systematically optimistic, and the consequence is a literature full of trials that were underpowered to detect the effects they were designed to find, or that required protocol amendments to survive long enough to reach their primary endpoint.
Where the Numbers Come From
A sample size calculation for a clinical trial requires three types of assumptions: an assumed effect size (the difference between treatment and control that the trial is powered to detect), a variance or event rate assumption (the variability in the outcome or the background event rate in the control arm), and assumptions about dropout, non-compliance, and loss to follow-up. In each category, the assumptions tend to be systematically optimistic.
Effect size assumptions are typically taken from pilot studies, single previous trials, or mechanistic rationale - sources that are prone to overestimation due to small sample sizes, winner's curse effects, and the natural tendency to plan for success rather than realistic outcomes. Variance and event rate assumptions come from historical data that may be outdated, drawn from different populations, or selected from among multiple available estimates based on which makes the calculation feasible within the available budget.
The Winner's Curse in Effect Size Estimation
The winner's curse is a well-documented phenomenon in statistics: when a finding first emerges from a study that was underpowered to detect true effects reliably, the observed effect size at the point of discovery is inflated. This happens because small studies only detect effects when chance sampling variation happens to push the estimate above the significance threshold. The true effect is smaller than the observed effect in the discovery study, but it is the discovery study estimate that gets used in the next trial's power calculation. The result is that the new trial is powered for an effect that is larger than the true effect, and is therefore underpowered for the actual outcome of the trial.
This dynamic plays out systematically across therapeutic areas. Meta-analyses consistently show that effect sizes decline as more and larger trials accumulate - the early dramatic findings give way to more modest but more reliable estimates. If the sample size for the follow-on trial was based on the initial dramatic finding, the follow-on trial will fail even when the treatment truly works.
Dropout and Loss to Follow-Up Assumptions
Dropout assumptions are almost universally optimistic. Most power calculations assume dropout rates that are lower than those actually observed in the trial. This matters because dropout reduces the effective sample size and, in certain missing data scenarios, biases the primary analysis. Trials that built in a 10% dropout allowance and experienced 25% dropout are common in the literature, and the consequence - an underpowered trial with a marginal primary endpoint result - is often attributed to the drug's biology rather than the calculation's assumptions.
The cure for optimistic dropout assumptions is empirical: use observed dropout rates from comparable trials in the same indication, in the same population, over similar follow-up durations. A systematic review of prior trials specifically to characterise dropout rates in the target indication is one of the highest-value applications of evidence synthesis in trial planning - and one of the least commonly conducted.
Internal Pilot Designs
An internal pilot design incorporates a pre-specified review of variance or event rate assumptions at an early interim point in the trial, with the option to revise the sample size if the observed values differ from those assumed. The key feature of a well-designed internal pilot is that the interim analysis is conducted on the nuisance parameters (variance, event rate) only, not on the treatment effect - the blinded sample size re-estimation preserves Type I error control in a way that unblinded re-estimation does not.
Internal pilots are underused. They protect against the most common source of trial failure - incorrect variance or event rate assumptions - without introducing the full complexity of an adaptive design, and they are accepted by regulatory agencies when pre-specified and clearly described in the protocol.
How a Better Evidence Review Changes the Calculation
The most direct intervention on sample size assumption quality is a rigorous systematic review of prior evidence before the power calculation is finalised. A review that identifies all trials in the relevant indication, extracts the control arm event rates or outcome variances, and synthesises them across studies provides a distribution of plausible assumptions rather than a single point estimate. Power calculations that use the median and the 25th percentile of this distribution - planning for a realistic scenario and a pessimistic scenario - produce trials that are more likely to succeed and more likely to report informative results regardless of outcome.
At NousLab, supporting the evidence review that informs trial design assumptions is one of the clearest applications of our work. The time invested in a rigorous review before the sample size is finalised is consistently worth it. See how NousLab supports trial design evidence reviews.
