Accounting for Student Attrition in Power Calculations: Benchmarks and Guidance

Rickles, Jordan; Zeiser, Kristina; West, Benjamin

doi:10.1080/19345747.2018.1502384

Cited by 9 publications

(17 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Attrition rates can also be obtained from prior research, for which power estimates can be adjusted accordingly. Thus, when analysing existing data or reporting results, documenting attrition rates will also help researchers to design experiments with greater precision (see Rickles et al, 2018).…”

Section: Report Attrition Rates and Standardized Variance Parametersmentioning

confidence: 99%

Statistical power and precision of experimental studies originated in the Republic of Turkey from 2010 to 2020: Current practices and some recommendations

Buluş

Koyuncu

2021

Participatory Educational Research

View full text Add to dashboard Cite

This study systematically reviews randomly selected 155 experimental studies in education field originated in the Republic of Turkey between 2010 and 2020. Indiscriminate choice of sample size in recent publications prompted us to evaluate their statistical power and precision. First, above and beyond our review, we could not identify any large-scale experiments such as cluster-randomized or multisite randomized trials, which overcome shortcomings of small-scale experiments, better suit to the organizational structure of the education field, nevertheless require far greater effort and financial resources. Second, none of the small-scale experiments has reported or conducted ex-ante power analysis. Third, results indicate that studies are sufficiently powered to detect medium effects and above (Cohen's d ≥ 0.50), however they are underpowered to detect small effects (Cohen's d ≤ 0.20). Trends in the past ten years indicate precision remained unchanged. We made several recommendations to increase the precision of experimental designs and improve their evidential values: Determine sample size prior to an experiment with power analysis routine, randomize subjects / clusters to obtain unbiased estimates, collect pre-test information and other relevant covariates, adjust for baseline differences beyond covariate control, document attrition, report standardized treatment effect and standardized variance parameters. Findings should be interpreted considering minimum effects in education that are relevant to education policy and practice.

show abstract

Section: Report Attrition Rates and Standardized Variance Parametersmentioning

confidence: 99%

Statistical power and precision of experimental studies originated in the Republic of Turkey from 2010 to 2020: Current practices and some recommendations

Buluş

Koyuncu

2021

Participatory Educational Research

View full text Add to dashboard Cite

show abstract

“…Following participant attrition, the final completer sample was modest (ACT n = 19, control n = 26). A risk of this level of dropout is that it could affect the integrity of the randomization and result in loss of power for detecting effects, allowing for a less realistic estimate of intervention impact (Rickles et al, 2017). Despite students often providing reasons for disengaging, factors such as nonacceptance of the intervention, no contact with the trainer postintervention, or low motivation to implement skills may have influenced dropout.…”

Section: Limitationsmentioning

confidence: 99%

Acceptance and Commitment Therapy in Group Format for College Students

Christodoulou

Flaxman

Lloyd

2021

J of College Counseling

View full text Add to dashboard Cite

We randomly assigned 71 student participants to an acceptance and commitment therapy (ACT) group training or to a wait list. All participants completed measures at preintervention, 1‐month postintervention, and 2‐month follow‐up. Students receiving ACT exhibited significantly reduced levels of general psychological distress and negative emotional symptoms at follow‐up. Mental health outcomes for ACT were mediated by increases in psychological flexibility and mindfulness. Results suggest that ACT group training could be an effective mental health intervention in educational settings.

show abstract

“…Many RCTs presumed the relative stability of their subjects. Yet empirical benchmarks based on longitudinal surveys place average national student mobility rates from 12% for a kindergarten to Grade 1 transition to 46% for a Grade 1 to Grade 5 transition, “[implying] that a study designed to follow students from Grade 1 to Grade 5 should anticipate losing half of the baseline sample if the study collected data only on students who remained in the study schools” (Rickles et al, 2018, p. 632).…”

Section: Reported Reasons For Null Findingsmentioning

confidence: 99%

“…For example, Lindsay et al’s (2017) evaluation of a college readiness support program reported 36% of all student subjects were missing a baseline Grade 8 standardized math test score in records provided by the participating schools and districts. Missing data are common, especially when collected during a different stage of schooling than the intervention or collected principally for a purpose unconnected to the study (Rickles et al, 2018). To offset these limitations, Rickles et al (2018) suggest that researchers explore whether the time and effort of a wider data collection strategy for extant information is warranted (such as collecting statewide standardized achievement test records in lieu of school- or district-wide) to reduce nonresponse rates.…”

Section: Reported Reasons For Null Findingsmentioning

confidence: 99%

“…Power analysis, for example, could incorporate empirical estimates of student and teacher mobility. Recent work by Rickles et al (2018) amends the standard formulas for minimum detectable effect sizes, adding parameters for overall and differential student attrition rates. The authors also calculate empirical benchmarks for student attrition during different grade transition periods using nationally representative, longitudinal surveys.…”

Section: Summary and Implicationsmentioning

confidence: 99%

See 1 more Smart Citation

Toward a Science of Failure Analysis: A Narrative Review

Allen-Platt

Gerstner

Boruch

et al. 2021

Review of Research in Education

View full text Add to dashboard Cite

When a researcher tests an educational program, product, or policy in a randomized controlled trial and detects a significant effect on an outcome, the intervention is usually classified as something that “works.” When expected effects are not found, there is seldom an orderly and transparent analysis of plausible reasons why. Accumulating and learning from possible failure mechanisms is not standard practice in education research, and it is not common to design interventions with causes of failure in mind. This chapter develops Boruch and Ruby’s proposition that the education sciences would benefit from a systematic approach to the study of failure. We review and taxonomize recent reports of large-scale randomized controlled trials in K–12 schooling that yielded at least one null or negative major outcome, including the nature of the event and reasons (if provided) for why it occurred. Our purpose is to introduce a broad framework for thinking about educational interventions that do not produce expected effects and seed a cumulative knowledge base on when, how, and why interventions do not reach expectations. The reasons why an individual intervention fails to elicit an outcome are not straightforward, but themes emerge when researchers’ reports are synthesized.

show abstract

Accounting for Student Attrition in Power Calculations: Benchmarks and Guidance

Cited by 9 publications

References 16 publications

Statistical power and precision of experimental studies originated in the Republic of Turkey from 2010 to 2020: Current practices and some recommendations

Statistical power and precision of experimental studies originated in the Republic of Turkey from 2010 to 2020: Current practices and some recommendations

Acceptance and Commitment Therapy in Group Format for College Students

Toward a Science of Failure Analysis: A Narrative Review

Contact Info

Product

Resources

About