Common Methodological Problems in Randomized Controlled Trials of Preventive Interventions

Steeger, Christine M.; Buckley, Pamela R.; Pampel, Fred C.; Gust, Charleen J.; Hill, Karl G.

doi:10.1007/s11121-021-01263-2

Cited by 29 publications

(27 citation statements)

References 62 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Blueprints for Healthy Youth Development ( https://www.blueprintsprograms.org/ ) provides an online registry of scientifically proven and scalable interventions that prevent or reduce the likelihood of antisocial behavior and promote a healthy course of youth development and adult maturity. Certified interventions are ranked as Promising, Model, or Model Plus based on the strength and extent of evaluation findings (Steeger et al, 2021 ) and readiness of the intervention to be disseminated with fidelity (Buckley et al, 2020 ). Model and Model Plus programs have demonstrated efficacy for changing outcomes over time and are recommended for large-scale implementation.…”

Section: Evidence-based Interventions and Registriesmentioning

confidence: 99%

“…Registries such as Blueprints advocate for implementing interventions that are known to work (see Steeger et al, 2021 ) and that can be implemented at scale with fidelity (see Buckley et al, 2020 ). Historically, Blueprints’ “Gold Standard” has been widespread adoption of Model or Model Plus interventions, implemented with fidelity, and sustained by routine funding sources within the community (Elliott et al, 2020 ).…”

Section: Balancing Fidelity Of Implementation and Adaptationmentioning

confidence: 99%

“…When adaptations consistent with the theoretical rationale for a program are deemed necessary, it is critical to establish the adaptation’s effect on outcomes with a high-quality randomized control trial (see Steeger et al, 2021 ) or methodologically sound meta-analysis (see Pigott & Polanin, 2020 ), which are two research designs that allow for causal inferences (Shadish et al, 2002 ; Weed, 2000 ; West & Thoemmes, 2010 ). To our knowledge, only a handful of studies have been completed to evaluate modifications to Blueprints-certified interventions on behavioral outcomes.…”

Section: Balancing Fidelity Of Implementation and Adaptationmentioning

confidence: 99%

See 2 more Smart Citations

Implementing Evidence-Based Preventive Interventions During a Pandemic

Buckley

Edwards

Ladika

et al. 2022

Glob Implement Res Appl

View full text Add to dashboard Cite

Section: Evidence-based Interventions and Registriesmentioning

confidence: 99%

Section: Balancing Fidelity Of Implementation and Adaptationmentioning

confidence: 99%

Section: Balancing Fidelity Of Implementation and Adaptationmentioning

confidence: 99%

See 1 more Smart Citation

Implementing Evidence-Based Preventive Interventions During a Pandemic

Buckley

Edwards

Ladika

et al. 2022

Glob Implement Res Appl

View full text Add to dashboard Cite

“…Important problems with the design, analysis and reporting quality of RCTs have been identified 10–12. Although excellent guidance about how to conduct RCTs is available,7 13 14 researchers conducting RCTs of complex interventions face challenges in designing and analysing trials that are not fully addressed in existing guidelines and are exacerbated by the belief common among scientists that guidance for trials of non-complex interventions should be strictly applied to trials of complex interventions.…”

Section: Introductionmentioning

confidence: 99%

Strengthening causal inference from randomised controlled trials of complex interventions

et al. 2022

View full text Add to dashboard Cite

Researchers conducting randomised controlled trials (RCTs) of complex interventions face design and analytical challenges that are not fully addressed in existing guidelines. Further guidance is needed to help ensure that these trials of complex interventions are conducted to the highest scientific standards while maximising the evidence that can be extracted from each trial. The key challenge is how to manage the multiplicity of outcomes required for the trial while minimising false positive and false negative findings. To address this challenge, we formulate three principles to conduct RCTs: (1) outcomes chosen should be driven by the intent and programme theory of the intervention and should thus be linked to testable hypotheses; (2) outcomes should be adequately powered and (3) researchers must be explicit and fully transparent about all outcomes and hypotheses before the trial is started and when the results are reported. Multiplicity in trials of complex interventions should be managed through careful planning and interpretation rather than through post hoc analytical adjustment. For trials of complex interventions, the distinction between primary and secondary outcomes as defined in current guidelines does not adequately protect against false positive and negative findings. Primary outcomes should be defined as outcomes that are relevant based on the intervention intent and programme theory, declared (ie, registered), and adequately powered. The possibility of confirmatory causal inference is limited to these outcomes. All other outcomes (either undeclared and/or inadequately powered) are secondary and inference relative to these outcomes will be exploratory.

show abstract

“…Federally supported evidence clearinghouses are influential because they affect billions of dollars of funding for social programs (e.g., through legislation like the Families First Preventive Services Act) (Garcia et al, 2020;Maynard, 2018), and they influence standards for conducting and reporting intervention research (Buckley et al, 2021;Steeger et al, 2021). Previous studies have examined characteristics of evidence clearinghouses that are important to stakeholders in prevention science and allied disciplines (Burkhardt et al, 2015;Davies & Silloway, 2016).…”

mentioning

confidence: 99%

Clearinghouse Standards of Evidence on the Transparency, Openness, and Reproducibility of Intervention Evaluations

2021

View full text Add to dashboard Cite

Clearinghouses are influential repositories of information on the effectiveness of social interventions. To identify which interventions are “evidence-based,” clearinghouses review intervention evaluations using published standards of evidence that focus primarily on internal validity and causal inferences. Open science practices can improve trust in evidence from evaluations on the effectiveness of social interventions. Including open science practices in clearinghouse standards of evidence is one of many efforts that could increase confidence in designations of interventions as “evidence-based.” In this study, we examined the policies, procedures, and practices of 10 federal evidence clearinghouses that review preventive interventions—an important and influential subset of all evidence clearinghouses. We found that seven consider at least one open science practice when evaluating interventions: replication (6 of 10 clearinghouses), public availability of results (6), investigator conflicts of interest (3), design and analysis transparency (3), study registration (2), and protocol sharing (1). We did not identify any policies, procedures, or practices related to analysis plan registration, data sharing, code sharing, material sharing, and citation standards. We provide a framework with specific recommendations to help federal and other evidence clearinghouses implement the Transparency and Openness Promotion (TOP) Guidelines. Our proposed “TOP Guidelines for Clearinghouses” includes reporting whether evaluations used open science practices, incorporating open science practices in their standards for receiving “evidence-based” designations, and verifying that evaluations used open science practices. Doing so could increase the trustworthiness of evidence used for policy making and support improvements throughout the evidence ecosystem.

show abstract

Common Methodological Problems in Randomized Controlled Trials of Preventive Interventions

Cited by 29 publications

References 62 publications

Implementing Evidence-Based Preventive Interventions During a Pandemic

Implementing Evidence-Based Preventive Interventions During a Pandemic

Strengthening causal inference from randomised controlled trials of complex interventions

Clearinghouse Standards of Evidence on the Transparency, Openness, and Reproducibility of Intervention Evaluations

Contact Info

Product

Resources

About