Design Parameters for Planning the Sample Size of Individual-Level Randomized Controlled Trials in Community Colleges

Abstract

The last two decades have seen a dramatic increase in randomized controlled trials (RCTs) conducted in community colleges. Yet, there is limited empirical information on the design parameters necessary to plan the sample size for RCTs in this context. We provide empirical estimates of key design parameters, discussing lessons based on the pattern of estimates across outcomes, semesters, and studies. Nearly all RCTs in community colleges use student random assignment within blocks (colleges or cohorts). Key design parameters for the minimum detectable true effect (MDTE) are the within-block outcome standard deviation [Formula: see text] and the within-block outcome variance explained by baseline covariates like student characteristics [Formula: see text]. The main analyses use student-level data from 8 to 14 RCTs including 5,649–7,099 students (depending on the outcome) with follow-up data for 3 years. A public database created for this paper includes parameter estimates for 30 RCTs including 65,637 students. Enrollment, credits earned, credential attainment, and grade point average. The within-block standard deviation [Formula: see text] and therefore the MDTE can be much larger in later semesters for enrollment outcomes and cumulative credits earned. There is substantial variation across studies in [Formula: see text] for degree attainment. Baseline covariates explain less than 10% of the variation in student outcomes. When planning the sample size for a study, researchers should be mindful of the follow-up period, use a range of values to calculate the MDTE for outcomes that vary across studies, and assume a value of [Formula: see text] between 0 and 0.05.

Empirical Benchmarks for Planning and Interpreting Causal Effects of Community College Interventions

“…Second, as we show in this paper, the distribution of effects across interventions can vary over time, with changing distributional means and variances. Moreover, as shown in Somers et al (2023), the standard deviation of the outcome also changes over time, among outcomes, and across interventions. Combining these facts, it may be quite difficult to make sense of comparisons of standardized effect sizes across time, outcomes, or evaluations.…”

“…The model does not control for students' baseline characteristics (like their gender and age). Doing so would not appreciably improve the precision of estimated effects because available characteristics are only weakly correlated with outcomes (Somers et al, 2023), and very few baseline variables are consistently available across studies.…”

“…Whereas for more comprehensive interventions, which tend to have larger effects , a larger minimum detectable true effect may be sufficient, depending on the goal of the study. Importantly, other design parameters are necessary to calculate the minimum detectable true effect of an intervention-for guidance specific to community college evaluations, see Somers et al (2023).…”

“…This article is one in a series of papers that capitalize on the unique dataset known as THE-RCT (for example, see Bailey & Weiss, 2022;Somers et al, 2023;. In addition to promoting increased learning through open and transparent data sharing, THE-RCT facilitates cross-study knowledge building.…”

“…With respect to planning, and more specifically power calculations, the "minimum detectable effect (MDE)" would more accurately be called the "minimum detectable true effect (MDTE)"-it is not the minimum detectable estimated effect (Somers et al, 2023). Consequently, when planning a study and considering whether the MDTE is of a magnitude that might be achieved by the intervention under study, a relevant benchmark is not "where does the MDTE lie on the distribution of estimated effects from past studies?"…”