The REPRISE Project: Protocol for an Evaluation of Reproducibility and Replicability in Syntheses of Evidence

Abstract

Background: Investigations of transparency, reproducibility and replicability in science have been directed largely at individual studies. It is just as critical to explore these issues in syntheses of studies, such as systematic reviews, given their influence on decision making and future research. We aim to explore various aspects relating to the transparency, reproducibility and replicability of several components of systematic reviews with meta-analysis of the effects of health, social, behavioural and educational interventions.Methods: The REPRISE (REProducibility and Replicability In Syntheses of Evidence) project consists of four studies. We will evaluate the completeness of reporting and sharing of review data, analytic code and other materials in a random sample of 300 systematic reviews of interventions published in 2020 (Study 1). We will survey authors of systematic reviews to explore their views on sharing review data, analytic code and other materials and their understanding of and opinions about replication of systematic reviews (Study 2). We will then evaluate the extent of variation in results when we (a) independently reproduce meta-analyses using the same computational steps and analytic code (if available) as used in the original review (Study 3), and (b) crowdsource teams of systematic reviewers to independently replicate a subset of methods (searches for studies, selection of studies for inclusion, collection of outcome data, and synthesis of results) in a sample of the original reviews; 30 reviews will be replicated by one team each and two reviews will be replicated by 15 teams (Study 4). Discussion: The REPRISE project takes a systematic approach to determine how reliable systematic reviews of interventions are. We anticipate that results of the REPRISE project will inform strategies to improve the conduct and reporting of future systematic reviews.

Data and code availability statements in systematic reviews of interventions were often missing or inaccurate: a content analysis

“…Furthermore, some systematic reviewers who use software which enables code sharing might not know how to share their code, feel uncomfortable with sharing the code due to concerns that errors might be detected in it or embarrassed about how it was written. As part of the REPRISE project (24), we plan to survey authors of systematic reviews to explore their views on sharing systematic review data and code, which should help elucidate this issue further.…”

“…This study arose out of the REPRISE (REProducibility and Replicability In Syntheses of Evidence) project (24). As part of REPRISE, we evaluated completeness of reporting in a random sample of 300 systematic reviews of interventions indexed during a four-week period in 2020.…”

“…The findings of the completeness of reporting evaluation will be reported elsewhere. Here, we provide a summary of the methods used to identify, select, and collect data from the systematic reviews; for full details see the published study protocol (24). In addition, we report the methods used to analyse the content of the data and code availability statements included in the systematic review reports.…”

Exploring Evidence Selection with the Inclusion Network

“…Systematic reviews have been widely adopted in a number of fields in the social sciences, ecological sciences, software engineering, and health sciences, in order to synthesize scholarly literature and provide guidance for practice. To limit bias and ensure the validity of their conclusions, systematic reviews typically use relatively standardized procedures, which have been customized and refined over decades in multiple fields (Chalmers et al, 2002;Cooper et al, 2019;Gurevitch et al, 2018;Haddaway et al, 2018;Okoli, 2015;Page, McKenzie, et al, 2021). Despite these attempts at standardization and reducing bias, it is common for systematic reviews published at the same time on the same topic to yield different conclusions (Gøtzsche, 1994;Jadad et al, 1997;Papatheodorou & Evangelou, 2022).…”

“…Selection of evidence consists of defining the selection principles (known as the "inclusion criteria") and exhaustively identifying relevant literature through a search and screening process. The evidence selection step is expected to find "everything" relevant on the topic, through a comprehensive search in multiple databases, with well-defined and consistently applied evidence selection criteria, often specified in advance in a protocol (Delaney & Tamás, 2018;Frampton et al, 2017;Lefebvre et al, 2019;Page et al, 2018;Page, McKenzie, et al, 2021;Rethlefsen et al, 2021). Evidence selection results in a list of publications, called the included primary study reports, which form a special subset of a systematic review's citations.…”