Stratification by quality induced selection bias in a meta-analysis of clinical trials

Abstract

Objective: The inconsistency demonstrated across strata when using different scales has been attributed to quality scores and stratification continues to be done using risk of bias domain judgments. This study examines if restricting primary meta-analyses to studies at low risk of bias or presenting meta-analyses stratified according to risk of bias is indeed the right approach to explore potential methodological bias. Study Design and Setting: Re-analysis of quality subgroups in an existing meta-analysis based on 25 different scales Results: We demonstrate that quality stratification itself is the problem because it induces a spurious association between effect size and precision within stratum. Studies with larger effects or lesser precision tend to be of lower quality-a form of collider stratification bias (stratum being the common effect of the reasons for these two outcomes) that leads to inconsistent results across scales. We also show that the extent of this association determines the variability in effect size and statistical significance across strata when conditioning on quality. Conclusions: We conclude that stratification by quality leads to a form of selection bias (collider stratification bias) and should be avoided. We demonstrate consistent results with an alternative method that includes all studies.

Letter re: stratification of meta-analyses based on risk of bias is appropriate and does not induce selection bias

“…We read, ''Stratification by quality induces selection bias in a meta-analysis of clinical trials'' by Stone et al [1] with interest. We think that the authors have misunderstood the implications of their findings and that their conclusions could mislead authors of systematic reviews.…”

“…They also observed small-study effectsda tendency for effects estimated in smaller studies to differ from those estimated in larger studies [4]dmore often in the lower-quality strata than in the higher-quality strata and attributed these results to the stratification (conditioning) on quality. They concluded that discrepancies between the pooled estimates in the high-and low-quality trials ''are largely because of conditioning on quality, which allows precision and effect size to associate within strata by quality'' [1].…”

“…By contrast, quality scales often include items that are not directly related to bias, such as ''Are both inclusion and exclusion criteria mentioned?,'' ''Was the therapeutic regimen adequately described for the treatment and control group,'' and ''Were the criteria for measuring outcomes clearly described?'' [1]. For this reason, we discourage the use of quality scales (including the composite scale developed by Stone et al) to appraise included studies and recommend authors focus on risk of bias.…”

“…Based on their findings, Stone et al recommend, ''The common strategy of assessing the impact of quality in meta-analysis by excluding lower or including higherquality studies should be abandoned'' [1]. However, a single old case study is not a good basis for drawing general recommendations for conduct of modern systematic reviews.…”

Quo Vadis, Cochrane Collaboration?

“…Their advice from their findings was “..that the use of summary scores to identify trials of high quality is problematic” (Jüni, Witschi, Bloch, & Egger, ). Twenty years later, another group of authoritative epidemiologists question why this practice is still maintained in Cochrane SRs since “stratification by quality leads to a form of selection bias, i.e., collider‐stratification bias, and should be avoided” in favor of other approaches (Stone et al, ).…”