Are missing data adequately handled in cluster randomised trials? A systematic review and guidelines

Díaz-Ordaz, Karla; Kenward, Michael G.; Cohen, Abie; Coleman, Claire L.; Eldridge, Sandra

doi:10.1177/1740774514537136

Cited by 53 publications

(69 citation statements)

References 41 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This review used a set of CRTs previously identified using a published electronic search strategy [27]. CRTs were eligible for inclusion if they were published in English in 2011.…”

Section: Methodsmentioning

confidence: 99%

“…They were excluded if they were quasi-experimental; were pilot, feasibility, or preliminary studies; did not collect any data at the individual level; only assessed cost-effectiveness; or were not the primary report of the trial findings. Trials were identified from PubMed using a pre-specified search strategy [27]. We randomly selected 100 of the 132 eligible trials for inclusion in this current review using randomly generated numbers in the statistical software package Stata.…”

Section: Methodsmentioning

confidence: 99%

See 1 more Smart Citation

Increased risk of type I errors in cluster randomised trials with small or medium numbers of clusters: a review, reanalysis, and simulation study

Kahan

Forbes

Ali

et al. 2016

Trials

View full text Add to dashboard Cite

BackgroundCluster randomised trials (CRTs) are commonly analysed using mixed-effects models or generalised estimating equations (GEEs). However, these analyses do not always perform well with the small number of clusters typical of most CRTs. They can lead to increased risk of a type I error (finding a statistically significant treatment effect when it does not exist) if appropriate corrections are not used.MethodsWe conducted a small simulation study to evaluate the impact of using small-sample corrections for mixed-effects models or GEEs in CRTs with a small number of clusters. We then reanalysed data from TRIGGER, a CRT with six clusters, to determine the effect of using an inappropriate analysis method in practice. Finally, we reviewed 100 CRTs previously identified by a search on PubMed in order to assess whether trials were using appropriate methods of analysis. Trials were classified as at risk of an increased type I error rate if they did not report using an analysis method which accounted for clustering, or if they had fewer than 40 clusters and performed an individual-level analysis without reporting the use of an appropriate small-sample correction.ResultsOur simulation study found that using mixed-effects models or GEEs without an appropriate correction led to inflated type I error rates, even for as many as 70 clusters. Conversely, using small-sample corrections provided correct type I error rates across all scenarios. Reanalysis of the TRIGGER trial found that inappropriate methods of analysis gave much smaller P values (P ≤ 0.01) than appropriate methods (P = 0.04–0.15). In our review, of the 99 trials that reported the number of clusters, 64 (65 %) were at risk of an increased type I error rate; 14 trials did not report using an analysis method which accounted for clustering, and 50 trials with fewer than 40 clusters performed an individual-level analysis without reporting the use of an appropriate correction.ConclusionsCRTs with a small or medium number of clusters are at risk of an inflated type I error rate unless appropriate analysis methods are used. Investigators should consider using small-sample corrections with mixed-effects models or GEEs to ensure valid results.

show abstract

“…This review used a set of CRTs previously identified using a published electronic search strategy [27]. CRTs were eligible for inclusion if they were published in English in 2011.…”

Section: Methodsmentioning

confidence: 99%

Section: Methodsmentioning

confidence: 99%

Increased risk of type I errors in cluster randomised trials with small or medium numbers of clusters: a review, reanalysis, and simulation study

Kahan

Forbes

Ali

et al. 2016

Trials

View full text Add to dashboard Cite

show abstract

“…Two recent reviews 6,96 indicate that missing outcome data is common in GRTs, though investigators frequently analyze only available data without accounting for the missing data pattern. When the covariate-dependent missingness (CDM) assumption is plausible, both mixed effects and GEE models provide unbiased estimates of the intervention effect when the CDM covariates are included in an analysis of all available data.…”

Section: Developments To Address Data Challengesmentioning

confidence: 99%

Review of Recent Methodological Developments in Group-Randomized Trials: Part 2—Analysis

et al. 2017

View full text Add to dashboard Cite

In 2004, Murray et al. published a review of methodological developments in both the design and analysis of group-randomized trials (GRTs). Over the last 13 years, there have been many developments in both areas. The goal of the current paper is to review developments in analysis, with a companion paper to focus on developments in design. As a pair, these papers update the 2004 review. This analysis paper includes developments in topics included in the earlier review, such as methods for parallel-arm GRTs, inference for conditional and marginal effects, and new topics including methods to account for multiple levels of clustering and alternative estimation methods such as augmented GEE, targeted maximum likelihood and quadratic inference functions. We also examine developments in dealing with missing outcome data, including doubly robust approaches, software available for analysis, and analysis of alternative group designs (including stepped wedge GRTs, network-randomized trials, pseudo-cluster randomized trials and individually-randomized group treatment trials). These alternative designs, like the parallel-arm GRT, require clustering to be accounted for in both their design and analysis.

show abstract

“…There are various approaches to imputation of missing posttest data to choose from (Díaz-Ordaz et al, 2014), depending on whether differential provider effects are taken into account (Taljaard, Donner, & Klar, 2008) plete pretest-to-posttest data). Consequently, the single-level regression-based approach diminishes differences between providers, whereas the multilevel approach takes these differences into account.…”

Section: Strengthsmentioning

confidence: 99%

Bias through selective inclusion and attrition: Representativeness when comparing provider performance with routine outcome monitoring data

Beurs

Warmerdam²,

Twisk

2019

Clin Psychology and Psychoth

View full text Add to dashboard Cite

Background Observational research based on routine outcome monitoring is prone to missing data, and outcomes can be biased due to selective inclusion at baseline or selective attrition at posttest. As patients with complete data may not be representative of all patients of a provider, missing data may bias results, especially when missingness is not random but systematic. Methods The present study establishes clinical and demographic patient variables relevant for representativeness of the outcome information. It applies strategies to estimate sample selection bias (weighting by inclusion propensity) and selective attrition bias (multiple imputation based on multilevel regression analysis) and estimates the extent of their impact on an index of provider performance. The association between estimated bias and response rate is also investigated. Results Provider‐based analyses showed that in current practice, the effect of selective inclusion was minimal, but attrition had a more substantial effect, biasing results in both directions: overstating and understating performance. For 22% of the providers, attrition bias was estimated to be in excess of 0.05 ES. Bias was associated with overall response rate (r = .50). When selective inclusion and attrition bring providers' response below 50%, it is more likely that selection bias increased beyond a critical level, and conclusions on the comparative performance of such providers may be misleading. Conclusions Estimates of provider performance were biased by selection, especially by missing data at posttest. Results on the extent and direction of bias and minimal requirements for response rates to arrive at unbiased performance indicators are discussed.

show abstract

Are missing data adequately handled in cluster randomised trials? A systematic review and guidelines

Cited by 53 publications

References 41 publications

Increased risk of type I errors in cluster randomised trials with small or medium numbers of clusters: a review, reanalysis, and simulation study

Increased risk of type I errors in cluster randomised trials with small or medium numbers of clusters: a review, reanalysis, and simulation study

Review of Recent Methodological Developments in Group-Randomized Trials: Part 2—Analysis

Bias through selective inclusion and attrition: Representativeness when comparing provider performance with routine outcome monitoring data

Contact Info

Product

Resources

About