The Problem With Subgroup Analyses in Clinical Trials
Subgroup analyses are statistically unreliable, frequently misleading, and clinically irresistible. Understanding why they produce false positives - and when they can be trusted - is essential for reading trial results critically.
The ISIS-2 trial, published in 1988, was one of the largest and most important cardiovascular trials of its era. It randomised more than seventeen thousand patients with suspected acute myocardial infarction and demonstrated that aspirin and streptokinase each reduced mortality, and that their combination was more effective than either alone. The results were robust and changed practice. They also included, apparently at the investigators' deliberate choice to illustrate a statistical point, a subgroup analysis showing that aspirin was ineffective - and possibly harmful - in patients born under the astrological signs of Libra or Gemini. The data supported this finding at nominal statistical significance. The point was made: if you test enough subgroups, chance will produce apparent effects in some of them.
Why Subgroup Analyses Produce False Positives
The logic is straightforward. If a trial tests its primary endpoint in the overall population with an alpha of 0.05, there is a 5% chance of a false positive. If the trial then tests the same endpoint in ten pre-specified subgroups, the probability that at least one subgroup analysis produces a false positive is approximately 40%. If the number of subgroups is larger, or if the subgroups were defined after looking at the data, the false positive rate rises further. The p-value threshold that was calibrated for a single test is being applied to what is functionally a multiple testing problem.
The multiple testing problem in subgroup analyses is not a subtle statistical concern - it means that a nominally significant subgroup finding in a trial with many subgroups has a very high probability of being false, regardless of how plausible it looks biologically.
The Interaction Test vs Within-Subgroup P-Values
The correct statistical approach to assessing whether a treatment effect differs across subgroups is the test for interaction (also called effect modification). The interaction test asks whether the treatment effect in subgroup A is statistically distinguishable from the treatment effect in subgroup B - a fundamentally different question from asking whether the treatment is effective within each subgroup separately. A trial that reports "treatment was effective in patients under 65 (p=0.03) but not in patients 65 and over (p=0.12)" is not demonstrating a subgroup difference - it is comparing one significant result to one non-significant result, which tells you nothing about whether the effects are truly different.
The interaction test has lower power than within-subgroup tests, which is why it is often non-significant even when within-subgroup p-values suggest a difference. This is not a failure of the test - it is the correct behaviour. Claiming a subgroup difference without a significant interaction test is claiming more than the data support.
Pre-Specification Requirements
Pre-specification of subgroup analyses - identifying them in the protocol before data collection - does not solve the multiple testing problem, but it substantially changes the inferential weight the findings deserve. A pre-specified subgroup analysis based on a specific biological hypothesis (patients with a particular genetic variant, patients above a specified disease severity threshold) is more credible than a post-hoc analysis of a subgroup that appeared interesting after the data were examined.
The CONSORT guidance on reporting subgroup analyses is explicit: the number of subgroup analyses performed should be stated, whether they were pre-specified should be stated, and the interaction test result should be reported for each. Studies that report only significant subgroup findings without this context are providing an incomplete - and potentially misleading - picture of the subgroup analysis exercise.
Famous Misleading Subgroups
The history of clinical trials is populated with subgroup findings that influenced practice and were subsequently refuted. The SYNTAX trial found that patients with simple coronary artery disease (low SYNTAX score) did as well with PCI as with CABG - a finding that shaped interventional cardiology for years before longer follow-up and larger datasets revealed the picture to be more complex. Early analyses of thrombolysis in stroke suggested benefit was confined to patients treated within three hours - a subgroup boundary that influenced guidelines and was later revised as more data accumulated.
In each case, the subgroup finding was biologically plausible, reported with apparent statistical support, and acted upon before adequate confirmatory evidence existed. This is the standard pattern: plausibility plus nominal significance, in the absence of pre-specification and interaction testing, is not sufficient evidence for a subgroup-specific treatment decision.
When Subgroup Findings Are Credible
Subgroup findings deserve more confidence when the interaction test is significant; when the subgroup was pre-specified with a clear biological rationale; when the direction and magnitude of effect are consistent with what biology and prior evidence predict; when the finding has been replicated across multiple independent trials; and when the subgroup is defined on a variable that is measured reliably and without ambiguity. Biomarker-defined subgroups in oncology - where molecular targets provide a clear mechanistic basis for differential response - come closest to meeting this standard, and they are the domain where subgroup-driven treatment decisions have the most defensible evidence base.
At NousLab, systematic reviews that include subgroup analyses are handled with careful attention to pre-specification status, interaction test reporting, and consistency across studies. A subgroup finding that appears in multiple trials, was pre-specified in each, and is supported by a significant interaction test in the majority is a different class of evidence from a single post-hoc finding. See how we grade subgroup evidence in NousLab reviews.
