Internal and external validity of cluster randomised trials: systematic review of recent trials

Abstract

Objectives To assess aspects of the internal validity of recently published cluster randomised trials and explore the reporting of information useful in assessing the external validity of these trials. Data extraction To assess aspects of internal validity we extracted data on appropriateness of sample size calculations and analyses, methods of identifying and recruiting individual participants, and blinding. To explore reporting of information useful in assessing external validity we extracted data on cluster eligibility, cluster inclusion and retention, cluster generalisability, and the feasibility and acceptability of the intervention to health providers in clusters. Results 21 (62%) trials accounted for clustering in sample size calculations and 30 (88%) in the analysis; about a quarter were potentially biased because of procedures surrounding recruitment and identification of patients; individual participants were blind to allocation status in 19 (56%) and outcome assessors were blind in 15 (44%). In almost half the reports, information relating to generalisability of clusters was poorly reported, and in two fifths there was no information about the feasibility and acceptability of the intervention. Conclusions Cluster randomised trials are essential for evaluating certain types of interventions. Issues affecting their internal validity, such as appropriate sample size calculations and analysis, have been widely disseminated and are now better addressed by researchers. Blinding of those identifying and recruiting patients to allocation status is recommended but is not always carried out. There may be fewer barriers to internal validity in trials in which individual participants are not recruited. External validity seems poorly addressed in many trials, yet is arguably as important as internal validity in judging quality as a basis for healthcare intervention. INTRODUCTIONIn cluster randomised trials, groups or clusters of individuals, rather than individuals themselves, are randomised. These trials are increasingly common in health services research, being particularly appropriate for evaluating interventions aimed at changing behaviour in patients or practitioners or changing organisation of services. Clusters might, for example, consist of patients in general practices or older people in nursing homes. Cluster randomised trials are pragmatic, measuring effectiveness rather than efficacy 1 and should therefore be both internally and externally valid.

Patient Outcomes in Simulation-Based Medical Education: A Systematic Review

“…Our findings regarding the prevalence of unit of analysis error mirror those in clinical medicine. [24][25][26][27][28][29] The overall MERSQI scores of this sample were slightly lower than those of 13 patient outcomes studies of resident shift length. 21 …”

“…22 Studies in which multiple patients contribute data for each trainee (patients clustered in trainees) require statistical techniques that account for such clustering. 23 Clinical research suggests that such unit of analysis errors are relatively common, ranging from 22 to 71 %, [24][25][26][27][28][29] and generally inflate the power of the analysis. 26,30 This may lead to conclusions of statistical significance when none are warranted.…”

A systematic review of the usage of flow diagram in cluster randomized trials

“…Ovi rezultati ukazuju na sve veću popularnost klaster randomizovanih studija što je u skladu sa zapažanjima drugih autora [14,15,1,16,17].…”

A primary care specialist genetics service: a cluster-randomised factorial trial

“…25 There was no bias in the selection of practices or patients to this trial since the primary outcomes were based on routine data, available for all the practices included in the analysis on an ITT basis. Recruitment of patients to the trial was effectively the primary outcome of the education comparison, referral rate.…”