An Empirical Study of Design Parameters for Assessing Differential Impacts for Students in Group Randomized Trials

Jaciw, Andrew P.; Лин, Ли; Ma, Boya

doi:10.2139/ssrn.2516005

Cited by 5 publications

(5 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Of note, model (3) is a direct extension of those studied in Raudenbush, 29 Li et al, 30 and Yang et al, 31 where only main effects of W i and X ij are considered. Further extensions of model (3) to allow for random coefficients for X ij can be found in Jaciw et al 32 and Dong et al 24 To proceed, we let W = E(W i ) denote the proportion of clusters that are randomized to the intervention group. When half of clusters are randomized to the intervention arm, W = 1∕2, but our results allow W to be any value in (0,1).…”

Section: Statistical Modelmentioning

confidence: 99%

See 1 more Smart Citation

Sample size requirements for detecting treatment effect heterogeneity in cluster randomized trials

Yang

Li²,

Starks

et al. 2020

Statistics in Medicine

View full text Add to dashboard Cite

Cluster randomized trials (CRTs) refer to experiments with randomization carried out at the cluster or the group level. While numerous statistical methods have been developed for the design and analysis of CRTs, most of the existing methods focused on testing the overall treatment effect across the population characteristics, with few discussions on the differential treatment effect among subpopulations. In addition, the sample size and power requirements for detecting differential treatment effect in CRTs remain unclear, but are helpful for studies planned with such an objective. In this article, we develop a new sample size formula for detecting treatment effect heterogeneity in two-level CRTs for continuous outcomes, continuous or binary covariates measured at cluster or individual level. We also investigate the roles of two intraclass correlation coefficients (ICCs): the adjusted ICC for the outcome of interest and the marginal ICC for the covariate of interest. We further derive a closed-form design effect formula to facilitate the application of the proposed method, and provide extensions to accommodate multiple covariates. Extensive simulations are carried out to validate the proposed formula in finite samples. We find that the empirical power agrees well with the prediction across a range of parameter constellations, when data are analyzed by a linear mixed effects model with a treatment-by-covariate interaction. Finally, we use data from the HF-ACTION study to illustrate the proposed sample size procedure for detecting heterogeneous treatment effects.

show abstract

Section: Statistical Modelmentioning

confidence: 99%

“…Of note, model () is a direct extension of those studied in Raudenbush, 29 Li et al, 30 and Yang et al, 31 where only main effects of W i and X ij are considered. Further extensions of model () to allow for random coefficients for X ij can be found in Jaciw et al 32 and Dong et al 24 …”

Section: Statistical Modelmentioning

confidence: 99%

Sample size requirements for detecting treatment effect heterogeneity in cluster randomized trials

Yang

Li²,

Starks

et al. 2020

Statistics in Medicine

View full text Add to dashboard Cite

show abstract

“…Newman et al (2012) found that a standardized effect size of .05 translates to 28 additional days of schooling in impacts on reading achievement). Reduced power to detect interaction effects (Jaciw et al, 2016a; Bloom, 2005b) may limit differences from reaching conventional levels of statistical significance among the Adherence measures.…”

Section: Resultsmentioning

confidence: 99%

Diagnosing bottlenecks in development-stage field experiments: Troubleshooting and finding opportunities for improvement

Jaciw

Лин

Schellinger

2018

Methodological Innovations

Self Cite

View full text Add to dashboard Cite

Randomized trials of social programs yield internally valid estimates of causal impacts on key outcomes. While estimates of gross impact serve as useful summaries, program developers require deeper information to drive improvement efforts, especially when no impact is observed. The first main goal of this work is to present a seven-stage diagnostic method for assessing process bottlenecks in experiments. Designed for programs still in development, the troubleshooting sequence uses mixed methods to assess where in a program's logic model the process is compromised. It includes post-experimental methods that are built into the design, to account for impact variation and test where effects are intensifying or diminishing. The second main purpose of this work is to demonstrate one such method in detail. The approach tests the relationship between fidelity of program implementation and impact. First, levels of achieved fidelity in the treatment group are modeled in terms of informative baseline covariates. The model is then used to index fidelity in both conditions. Informed only by pre-randomization characteristics of individuals, the model-based fidelity scores are unbiased by endogeneity, and allow assessment of whether impacts on key outcomes vary by levels of fidelity. Results can help program developers focus improvement efforts. We illustrate the seven-step diagnostic process through a randomized trial of the Internet-Based Reading Apprenticeship Improving Science Education (iRAISE) program. Eighty-two high school science teachers and 1468 students were randomly assigned to a literacy program or control. There was no overall impact on achievement. Applying the diagnostic process revealed this was not due to a weak program contrast between conditions, or an inadequate assessment; rather, lower-than-expected impact was likely due to weaker than intended implementation.

show abstract

“…Evidence for large interactions is weak. Few studies have sufficient power to precisely estimate interactions, but those that do often fail to detect any subgroup effects (e.g., Collins et al, 2016; Jaciw, Lin, & Ma, 2016; Michalopoulos, Schwartz, & Adams-Ciardullo, 2000). 6 Furthermore, what estimates we have of subgroups are likely in part due to common differential impacts by subgroups not to true interactions.…”

Section: An Eb Approachmentioning

confidence: 99%

Editor in Chief’s Comment: External Validity in Systematic Reviews

Klerman

2017

Eval Rev

View full text Add to dashboard Cite

An Empirical Study of Design Parameters for Assessing Differential Impacts for Students in Group Randomized Trials

Cited by 5 publications

References 16 publications

Sample size requirements for detecting treatment effect heterogeneity in cluster randomized trials

Sample size requirements for detecting treatment effect heterogeneity in cluster randomized trials

Diagnosing bottlenecks in development-stage field experiments: Troubleshooting and finding opportunities for improvement

Editor in Chief’s Comment: External Validity in Systematic Reviews

Contact Info

Product

Resources

About