Optimal Design of Cluster- and Multisite-Randomized Studies Using Fallible Outcome Measures

Abstract

Background: Evaluation studies frequently draw on fallible outcomes that contain significant measurement error. Ignoring outcome measurement error in the planning stages can undermine the sufficiency and efficiency of an otherwise well-designed study and can further constrain the evidence studies bring to bear on the effectiveness of programs. Objectives: We develop simple formulas to adjust statistical power, minimum detectable effect (MDE), and optimal sample allocation formulas for two-level cluster- and multisite-randomized designs when the outcome is subject to measurement error. Results: The resulting adjusted formulas suggest that outcome measurement error typically amplifies treatment effect uncertainty, reduces power, increases the MDE, and undermines the efficiency of conventional optimal sampling schemes. Therefore, achieving adequate power for a given effect size will typically demand increased sample sizes when considering fallible outcomes, while maintaining design efficiency will require increasing portions of a budget be applied toward sampling a larger number of individuals within clusters. We illustrate evaluation planning with the new formulas while comparing them to conventional formulas using hypothetical examples based on recent empirical studies. To encourage adoption of the new formulas, we implement them in the R package PowerUpR and in the PowerUp software.

Power Analysis for Two-Level Multisite Randomized Cost-Effectiveness Trials

“…One limitation of the current study is that we assume the cost data are measured with validity and reliability through the ingredients method, while some ingredients might be unobserved, unintended, hard to identify, or subject to measurement error. Prior studies (e.g., Cox & Kelcey, 2019) have discussed the impact of the measurement error of the effectiveness measures (e.g., test scores) on power and MDES; however, to date there have not been studies to address the validity of the ingredient methods or the reliability (or measurement error) of the cost estimates–based ingredient methods. When cost estimates are subject to measurement error, the uncertainty (e.g., SE s) of the estimated incremental cost (i.e., Δ C ) will increase, and thus the power to detect the cost-effectiveness of one particular intervention would decrease and the MDES would increase.…”

Statistical power and precision of experimental studies originated in the Republic of Turkey from 2010 to 2020: Current practices and some recommendations

“…Many more groups need to be randomly assigned to treatment and control conditions to have adequate power, also known as cluster-randomization in the literature. There are many scholars spearheading this line of research, and guide practitioners to design rigorous clusterrandomized trials (e.g., Bloom, 1995Bloom, , 2006Bloom et al, 1999;Bulus & Dong, 2021;Bulus & Şahin, 2019;Cox & Kelcey, 2019a, 2019bDong, Kelcey, & Spybrook, 2017;Dong & Maynard, 2013;Dong, Kelcey, & Spybrook, 2017;Kelcey, Dong, Spybrook, & Cox, 2017;Kelcey, Dong, Spybrook, & Shen, 2017;Konstantopoulos, 2009Konstantopoulos, , 2011Konstantopoulos, , 2013Raudenbush, 1997;Raudenbush & Liu, 2000;and many others). There are publicly available software tools that implement results from these studies to assist with the design of cluster-randomized trials (e.g., PowerUp!, Dong & Maynard, 2013;PowerUpR, Bulus et al, 2019;OD+, Spybrook et al, 2011).…”