The REPRISE project: protocol for an evaluation of REProducibility and Replicability In Syntheses of Evidence

Abstract

Background Investigations of transparency, reproducibility and replicability in science have been directed largely at individual studies. It is just as critical to explore these issues in syntheses of studies, such as systematic reviews, given their influence on decision-making and future research. We aim to explore various aspects relating to the transparency, reproducibility and replicability of several components of systematic reviews with meta-analysis of the effects of health, social, behavioural and educational interventions. Methods The REPRISE (REProducibility and Replicability In Syntheses of Evidence) project consists of four studies. We will evaluate the completeness of reporting and sharing of review data, analytic code and other materials in a random sample of 300 systematic reviews of interventions published in 2020 (Study 1). We will survey authors of systematic reviews to explore their views on sharing review data, analytic code and other materials and their understanding of and opinions about replication of systematic reviews (Study 2). We will then evaluate the extent of variation in results when we (a) independently reproduce meta-analyses using the same computational steps and analytic code (if available) as used in the original review (Study 3), and (b) crowdsource teams of systematic reviewers to independently replicate a subset of methods (searches for studies, selection of studies for inclusion, collection of outcome data, and synthesis of results) in a sample of the original reviews; 30 reviews will be replicated by 1 team each and 2 reviews will be replicated by 15 teams (Study 4). Discussion The REPRISE project takes a systematic approach to determine how reliable systematic reviews of interventions are. We anticipate that results of the REPRISE project will inform strategies to improve the conduct and reporting of future systematic reviews.

Changing patterns in reporting and sharing of review data in systematic reviews with meta-analysis of the effects of interventions: a meta-research study

“…Some studies have explored these barriers in general academia (40,41), but we are uncertain whether researchers in evidence synthesis will face all of these barriers or even unidentified barriers unique to systematic reviews and meta-analyses. Future studies in the REPRISE project will explore systematic reviewers’ perspectives on barriers and incentives to reporting and data-sharing in order to address these questions (25).…”

“…This study was conducted as one of a suite of studies in the REPRISE (REProducibility and Replicability In Syntheses of Evidence) project. The REPRISE project is investigating various aspects relating to the transparency, reproducibility and replicability of systematic reviews with meta-analysis of the effects of health, social, behavioural and educational interventions (25). Methods for all studies were pre-specified in the same protocol (25).…”

Fear of COVID-19 and its association with mental health-related factors: systematic review and meta-analysis

“…28 The study's findings are reported according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines. 29…”

TRIPS to Where? A Narrative Review of the Empirical Literature on Intellectual Property Licensing Models to Promote Global Diffusion of Essential Medicines

“…The rationale for opting for a narrative review in this study is to outline what has been previously published on the topic and seek new study areas not yet explored [50,51]. Nonetheless, each step of the review was informed by the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) reporting guidelines [52,53].…”

“…Additional gray literature and scholarly materials, including working papers, dissertations, and book sections, were manually searched using institutional websites, Google Scholar, and snowballing through review of references in the identified publications. Full information on MeSH terms, keywords, and Boolean operators used and search strategy developed for each database is provided in the Supplementary Materials (Table S1) to improve reproducibility of results [52]. All publications that reported results from an original quantitative or qualitative research study on the impact of voluntary IP models on indicators of access to medicines were included.…”