Clearinghouse Standards of Evidence on the Transparency, Openness, and Reproducibility of Intervention Evaluations

Abstract

Clearinghouses are influential repositories of information on the effectiveness of social interventions. To identify which interventions are “evidence-based,” clearinghouses review intervention evaluations using published standards of evidence that focus primarily on internal validity and causal inferences. Open science practices can improve trust in evidence from evaluations on the effectiveness of social interventions. Including open science practices in clearinghouse standards of evidence is one of many efforts that could increase confidence in designations of interventions as “evidence-based.” In this study, we examined the policies, procedures, and practices of 10 federal evidence clearinghouses that review preventive interventions—an important and influential subset of all evidence clearinghouses. We found that seven consider at least one open science practice when evaluating interventions: replication (6 of 10 clearinghouses), public availability of results (6), investigator conflicts of interest (3), design and analysis transparency (3), study registration (2), and protocol sharing (1). We did not identify any policies, procedures, or practices related to analysis plan registration, data sharing, code sharing, material sharing, and citation standards. We provide a framework with specific recommendations to help federal and other evidence clearinghouses implement the Transparency and Openness Promotion (TOP) Guidelines. Our proposed “TOP Guidelines for Clearinghouses” includes reporting whether evaluations used open science practices, incorporating open science practices in their standards for receiving “evidence-based” designations, and verifying that evaluations used open science practices. Doing so could increase the trustworthiness of evidence used for policy making and support improvements throughout the evidence ecosystem.

Implementing Evidence-Based Preventive Interventions During a Pandemic

“…First, respondents were limited to those interventions certified by the Blueprints registry. Many other interventions are listed on other registries and/or are being broadly implemented (Axford et al, 2022 ; Burkhardt et al, 2015 ; Means et al, 2015 , Mayo-Wilson et al, 2021 ). Second, among Blueprints certified interventions, not all intervention developers or purveyors responded.…”

“…Given the increasing volume and scientific complexity of literature evaluating the efficacy and effectiveness of interventions (Bastian et al, 2010 ; Gottfredson et al, 2015 ), these registries make the evaluation literature more accessible to practitioners and raise awareness about the existence of evidence-based interventions. The importance of registries acting as intermediaries between researchers and program users is evident in the large number that exist—up to 24 across the United States and Europe alone (Axford et al, 2022 ; Burkhardt et al, 2015 ; Means et al, 2015 ), ten of which are currently funded by the United States federal government (Mayo-Wilson et al, 2021 ).…”

Exploring enablers and barriers to implementing the Transparency and Openness Promotion Guidelines: a theory-based survey of journal editors

“…In a previous study, we identified 10 federal evidence clearinghouses that rate the quality of evidence concerning the effectiveness of social interventions [ 75 ]. We then identified 339 journals that published at least one intervention report that a clearinghouse used to give its highest rating for quality of evidence [ 76 ].…”

Recruter de meilleurs échantillons en éducation

“…L'utilisation du test pourrait par conséquent entrainer un surdiagnostic de difficultés en mathématiques chez les élèves de cinquième et sixième année. Puisque la Lorsqu'une étude génère des résultats inattendus et potentiellement importants, les méthodologues recommandent entres autres de reproduire ces résultats auprès d'un nouvel échantillon avant de proposer des modifications aux théories, pratiques, programmes ou politiques (ex : Duncan, Engel, Claessens et Dowsett, 2014;Mayo-Wilson, Grant et Supplee, 2021)…”