Choosing an imbalance metric for covariate-constrained randomization in multiple-arm cluster-randomized trials

Abstract

Background In cluster-randomized controlled trials (C-RCTs), covariate-constrained randomization (CCR) methods efficiently control imbalance in multiple baseline cluster-level variables, but the choice of imbalance metric to define the subset of “adequately balanced” possible allocation schemes for C-RCTs involving more than two arms and continuous variables is unclear. In an ongoing three-armed C-RCT, we chose the min(three Kruskal–Wallis [KW] test P values) > 0.30 as our metric. We use simulation studies to explore the performance of this and other metrics of baseline variable imbalance in CCR. Methods We simulated three continuous variables across three arms under varying allocation ratios and assumptions. We compared the performance of min(analysis of variance [ANOVA] P value) > 0.30, min(KW P value) > 0.30, multivariate analysis of variance (MANOVA) P value > 0.30, min(nine possible t test P values) > 0.30, and min(Wilcoxon rank-sum [WRS] P values) > 0.30. Results Pairwise comparison metrics ( t test and WRS) tended to be the most conservative, providing the smallest subset of allocation schemes (10%–13%) meeting criteria for acceptable balance. Sensitivity of the min( t test P values) > 0.30 for detecting non-trivial imbalance was 100% for both hypothetical and resampled simulation scenarios. The KW criterion maintained higher sensitivity than both the MANOVA and ANOVA criteria (89% to over 99%) but was not as sensitive as pairwise criteria. Conclusions Our criterion, the KW P value > 0.30, to signify “acceptable” balance was not the most conservative, but it appropriately identified imbalance in the majority of simulations. Since all are related, CCR algorithms involving any of these imbalance metrics for continuous baseline variables will ensure robust simultaneous control over multiple continuous baseline variables, but we recommend care in determining the threshold of “acceptable” levels of (im)balance. Trial registration This trial is registered on ClinicalTrials.gov (initial post: December 1, 2016; identifier: NCT02979444 ). Electronic supplementary material The online version of this article (10.1186/s13063-019-3324-5) contains supplementary material, which is available to authorized users.

Design and analysis of three‐arm parallel cluster randomized trials with small numbers of clusters

“…Little analysis exists of balancing randomization methods in the multi‐arm cluster trial setting and there does not exist any extension and evaluation of balancing randomization methods involving matching to a multi‐arm setting. Ciolino et al 22 consider covariate‐constrained randomization in a multi‐arm trial setting and demonstrate that it does indeed improve balance between trial arms, but the consequences of this on inferences are not explored.…”

Application of randomization techniques for balancing site covariates in the adult day service plus pragmatic cluster-randomized trial

“…Covariate-constrained randomization techniques are increasingly popular in CRTs, but uncertainties remain over the specifics of implementation, how overall imbalance should be assessed, and what constitutes acceptable balance thresholds [ 15 ]. Stratification is sometimes used in CRTs [ 20 ], but this method can become unwieldy when balance is desired on more than just a few site characteristics [ 1 , 3 ].…”

“…In addition to overall imbalance control targets across the site covariates collectively, the degree of imbalance observed on individual site characteristics might also be examined and considered. Ciolino and colleagues [ 15 ] conducted simulation studies and endorsed a threshold of having all p-values from nonparametric analyses of individual site characteristics being 0.30 or higher as indicative of acceptable balance. However, p-values are largely dependent on the number of sites being randomized, making this threshold easier to achieve with some notable imbalances still remaining when the number of sites to be randomized is relatively small.…”

“…We recommend that future applications of optimized randomization procedures for CRTs set a priori targets for controlling imbalance and use standardized metrics such as H. Such targets might also be supplemented with realistic criteria for individual balancing factors. For example, one could specify that the AVDMs for individual site characteristics be below a pre-specified cut point, such as the 25th percentile, that p-values for intervention vs. control comparisons of individual balancing factors all exceed 0.30 [ 15 ], and that only randomizations with H below the 10th percentile be considered as acceptable. For the 6 balancing variables that represented the 5 site characteristics in the ADS Plus trial, the 10th percentile corresponds to H = 0.48, and 4 of the initial 20 randomizations would have been considered acceptable.…”

“…When considering different optimized randomization approaches for CRTs, suitable metrics of covariate imbalance and conventional benchmarks on what is considered to be acceptable balance would be useful [ 15 ]. Raab and Butcher [ 16 ] proposed a balance criterion (B) that consisted of a weighted sum of squared mean differences for site variables.…”