An Empirical Study of Design Parameters for Assessing Differential Impacts for Students in Group Randomized Trials

Jaciw, Andrew P.; Лин, Ли; Ma, Boya

doi:10.1177/0193841x16659600

Cited by 10 publications

(2 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In the last decade and a half, great strides have been made to generate methodological tools and benchmarks to support the design and planning of RCT evaluations in K-12 education. K-12 researchers now have available estimates of the design parameters necessary for conducting statistical power analyses (e.g., see: Bloom, Zhu et al, 2008 ; Deke et al, 2010 ; Dong et al, 2016 , 2021 ; Hedberg, 2016 ; Hedges & Hedberg, 2007a , 2007b , 2014 ; Jacob et al, 2010 ; Kelcey et al, 2016 ; Jaciw et al, 2016 ; Juras, 2016 ; Schochet, 2008 ; Weiss et al, 2017 ; Westine et al, 2013 ; Xu & Nichols, 2010 ).…”

Section: Introductionmentioning

confidence: 99%

Design Parameters for Planning the Sample Size of Individual-Level Randomized Controlled Trials in Community Colleges

2022

View full text Add to dashboard Cite

The last two decades have seen a dramatic increase in randomized controlled trials (RCTs) conducted in community colleges. Yet, there is limited empirical information on the design parameters necessary to plan the sample size for RCTs in this context. We provide empirical estimates of key design parameters, discussing lessons based on the pattern of estimates across outcomes, semesters, and studies. Nearly all RCTs in community colleges use student random assignment within blocks (colleges or cohorts). Key design parameters for the minimum detectable true effect (MDTE) are the within-block outcome standard deviation [Formula: see text] and the within-block outcome variance explained by baseline covariates like student characteristics [Formula: see text]. The main analyses use student-level data from 8 to 14 RCTs including 5,649–7,099 students (depending on the outcome) with follow-up data for 3 years. A public database created for this paper includes parameter estimates for 30 RCTs including 65,637 students. Enrollment, credits earned, credential attainment, and grade point average. The within-block standard deviation [Formula: see text] and therefore the MDTE can be much larger in later semesters for enrollment outcomes and cumulative credits earned. There is substantial variation across studies in [Formula: see text] for degree attainment. Baseline covariates explain less than 10% of the variation in student outcomes. When planning the sample size for a study, researchers should be mindful of the follow-up period, use a range of values to calculate the MDTE for outcomes that vary across studies, and assume a value of [Formula: see text] between 0 and 0.05.

show abstract

Section: Introductionmentioning

confidence: 99%

Design Parameters for Planning the Sample Size of Individual-Level Randomized Controlled Trials in Community Colleges

2022

View full text Add to dashboard Cite

show abstract

“…In conjunction with the increasing emphasis on moderated treatment effects is a growing literature detailing design and analysis techniques that support the inclusion of moderator variables. For example, existing research details the inclusion and analysis of different types of moderator variables (e.g., categorical or continuous) in various experimental study designs (Bloom, 2005; Dong et al, 2018; Jaciw et al, 2016; Spybrook et al, 2016). Many of these advancements have been implemented in software (e.g., Dong et al, 2016), expanding the capacity of evaluators to plan for and capture treatment effect moderation.…”

mentioning

confidence: 99%

Statistical Power for Detecting Moderation in Partially Nested Designs

Cox

Kelcey

2022

American Journal of Evaluation

View full text Add to dashboard Cite

Analysis of the differential treatment effects across targeted subgroups and contexts is a critical objective in many evaluations because it delineates for whom and under what conditions particular programs, therapies or treatments are effective. Unfortunately, it is unclear how to plan efficient and effective evaluations that include these moderated effects when the design includes partial nesting (i.e., disparate grouping structures across treatment conditions). In this study, we develop statistical power formulas to identify requisite sample sizes and guide the planning of evaluations probing moderation under two-level partially nested designs. The results suggest that the power to detect moderation effects in partially nested designs is substantially influenced by sample size, moderation effect size, and moderator variance structure (i.e., varies within groups only or within and between groups). We implement the power formulas in the R-Shiny application PowerUpRShiny and demonstrate their use to plan evaluations.

show abstract

Sample size requirements for detecting treatment effect heterogeneity in cluster randomized trials

Yang

Li²,

Starks

et al. 2020

Statistics in Medicine

View full text Add to dashboard Cite

Cluster randomized trials (CRTs) refer to experiments with randomization carried out at the cluster or the group level. While numerous statistical methods have been developed for the design and analysis of CRTs, most of the existing methods focused on testing the overall treatment effect across the population characteristics, with few discussions on the differential treatment effect among subpopulations. In addition, the sample size and power requirements for detecting differential treatment effect in CRTs remain unclear, but are helpful for studies planned with such an objective. In this article, we develop a new sample size formula for detecting treatment effect heterogeneity in two-level CRTs for continuous outcomes, continuous or binary covariates measured at cluster or individual level. We also investigate the roles of two intraclass correlation coefficients (ICCs): the adjusted ICC for the outcome of interest and the marginal ICC for the covariate of interest. We further derive a closed-form design effect formula to facilitate the application of the proposed method, and provide extensions to accommodate multiple covariates. Extensive simulations are carried out to validate the proposed formula in finite samples. We find that the empirical power agrees well with the prediction across a range of parameter constellations, when data are analyzed by a linear mixed effects model with a treatment-by-covariate interaction. Finally, we use data from the HF-ACTION study to illustrate the proposed sample size procedure for detecting heterogeneous treatment effects.

show abstract

An Empirical Study of Design Parameters for Assessing Differential Impacts for Students in Group Randomized Trials

Cited by 10 publications

References 18 publications

Design Parameters for Planning the Sample Size of Individual-Level Randomized Controlled Trials in Community Colleges

Design Parameters for Planning the Sample Size of Individual-Level Randomized Controlled Trials in Community Colleges

Statistical Power for Detecting Moderation in Partially Nested Designs

Sample size requirements for detecting treatment effect heterogeneity in cluster randomized trials

Contact Info

Product

Resources

About