The methodological and reporting characteristics of Campbell reviews: A systematic review

Wang, Xiaoqin; Welch, Vivian; Li, Meixuan; Yao, Liang; Littell, Julia H.; Li, Huijuan; Yang, Nan; Wang, Jianjian; Shamseer, Larissa; Chen, Yaolong; Yang, Kehu; Grimshaw, Jeremy

doi:10.1002/cl2.1134

Cited by 15 publications

(15 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We decided to just document and not exclude articles without a protocol. It is not clear why some systematic reviews, with authors deciding to adopt the rigorous methodological approaches of, for example, Campbell Collaboration's MECCIR (Wang et al, 2021) or similar, do not have a pre-registered protocol published in an open repository for others to scrutinize. This is one area, we think, that there is room for improvement.…”

Section: Discussionmentioning

confidence: 99%

“…The method also includes that the search should have identified all relevant studies that match the study eligibility criteria and steps taken to ensure appropriate quality or assess quality or bias in included studies, both within studies (e.g., limitations in method) and between studies (e.g., publication bias), as well as reproducible workflows (e.g., searches and synthesis). In similarity, the Campbell Collaboration uses an adapted form called MECCIR, Methodological Expectations of Campbell Collaboration Intervention Reviews (Wang et al, 2021). Both organizations guide review teams to apply 'best practice' throughout the review.…”

Section: Standards In Systematic Reviewsmentioning

confidence: 99%

See 1 more Smart Citation

Risk of bias and open science practices insystematic reviews of educational effectiveness:A meta-review

Nordström¹,

Kalmendal²,

Batinovic³

2022

Preprint

View full text Add to dashboard Cite

BackgroundObjectives: In order to produce the most reliable syntheses of the effectiveness of educational interventions, systematic reviews need to adhere to rigorous methodological standards. This meta-review investigated risk of bias and presence of open science practices like data sharing and reproducibility of the review procedure, in recently published reviews in education. MethodsEligibility criteria: We included all systematic reviews of educational interventions, instructions and methods for all K-12 student populations in any school form with experimental or quasi-experimental designs (an active manipulation of the intervention) with comparisons and where the outcome variables were academic performance of any kind. Information sources: We searched the database Education Resources Information Center (ERIC) through the years 2019-2021. In parallel we hand-searched four major educational review journals for systematic reviews: Educational research review (Elsevier), Educational review (Taylor & Francis), Review of Education (Wiley) and Review of educational research (AERA).Risk of bias: Systematic reviews were formally assessed by the tool ROBIS - Risk of bias in systematic reviews.Synthesis of results: Systematic reviews were assessed with the risk of bias tool ROBIS and whether the studies had pre-registered protocols, shared primary research data, and if a third party could reproduce search strings and details of where exactly primary research data was extracted from. ResultsIncluded studies: From a total of 88 studies that matched our PICOS, 18 systematic reviews passed a shortened ROBIS assessment and were subjected to the full ROBIS assessment.Synthesis of results: 10 educational systematic reviews were judged as low risk of bias from a total of 88 systematic reviews. This means that approximately 11 percent of the systematic reviews were of low risk of bias. The rest were classified as high risk of bias during the shortened ROBIS or assessed as high risk or unclear risk of bias following the full ROBIS assessment. Of the 10 low risk of bias reviews, 6 had detailed their search sufficiently enough for a third party to reproduce, 3 reviews shared the data from primary studies, however none had specified how and from where exactly data from primary studies were extracted.DiscussionInterpretation: The study shows that at least a small part of systematic reviews in education are of high quality, as shown by being assessed as low risk of bias, but most systematic reviews in this area are of lower quality. There are still improvements in this field to be expected as even the low risk of bias reviews are not consistent regarding pre-registered protocols, data sharing, reproducibility of primary research data and reproducible search strings. Funding: Swedish Research Council, 2020-03430Registration: https://doi.org/10.17605/OSF.IO/WXFDJ

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Standards In Systematic Reviewsmentioning

confidence: 99%

Risk of bias and open science practices insystematic reviews of educational effectiveness:A meta-review

Nordström¹,

Kalmendal²,

Batinovic³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…In a methodological review of 96 systematic reviews published in the journal, Campbell Systematic Reviews a total of 84 reviews included citation chasing methods 15 . The percentage of reviews including the method increased from 80% to 94% for January 2011 to September 2014 relative to October 2014 to February 2018.…”

Section: Introductionmentioning

confidence: 99%

Citationchaser: A tool for transparent and efficient forward and backward citation chasing in systematic searching

Haddaway

Grainger

Gray

2022

Research Synthesis Methods

192

109

View full text Add to dashboard Cite

Systematic searching aims to find all possibly relevant research from multiple sources, the basis for an unbiased and comprehensive evidence base. Along with bibliographic databases, systematic reviewers use a variety of additional methods to minimise procedural bias. Citation chasing exploits connections between research articles to identify relevant records for a review by making use of explicit mentions of one article within another. Citation chasing is a popular supplementary search method because it helps to build on the work of primary research and review authors. It does so by identifying potentially relevant studies that might otherwise not be retrieved by other search methods;for example, because they did not use the review authors' search terms in the specified combinations in their titles, abstracts, or keywords. Here, we briefly provide an overview of citation chasing as a method for systematic reviews.Furthermore, given the challenges and high resource requirements associated with citation chasing, the limited application of citation chasing in otherwise rigorous systematic reviews, and the potential benefit of identifying terminologically disconnected but semantically linked research studies, we have developed and describe a free and open source tool that allows for rapid forward and backward citation chasing. We introduce citationchaser, an R package and Shiny app for conducting forward and backward citation chasing from a starting set of articles. We describe the sources of data, the backend code functionality, and the user interface provided in the Shiny app.

show abstract

“…Campbell Collaboration and Cochrane reviews are expected to meet more rigorous standards. The methodological quality and reporting characteristics of Campbell Collaboration reviews have improved over time, but only 17% of a sample of 96 Campbell reviews were assessed as high quality [ 2 ].…”

mentioning

confidence: 99%

“…Their guidelines state: “Every Campbell review is required to have clear criteria for eligible research, an explicit and comprehensive search strategy, systematic and replicable coding and analysis of the key features and findings of the studies reviewed, and an integrative summary of those findings” ([ 3 ] p. 5). In 2014, the Campbell Collaboration established an explicit set of Methodological Expectations for Campbell Collaboration Intervention Reviews to guide the conduct and reporting of its reviews [ 2 , 4 , 5 ]. Each MECCIR standard is identified by a number, following C for conduct or R for reporting.…”

mentioning

confidence: 99%

The Campbell Collaboration’s systematic review of school-based anti-bullying interventions does not meet mandatory methodological standards

Littell

Gorman

2022

Syst Rev

View full text Add to dashboard Cite

Background Many published reviews do not meet the widely accepted PRISMA standards for systematic reviews and meta-analysis. Campbell Collaboration and Cochrane reviews are expected to meet even more rigorous standards, but their adherence to these standards is uneven. For example, a newly updated Campbell systematic review of school-based anti-bullying interventions does not appear to meet many of the Campbell Collaboration’s mandatory methodological standards. Issues In this commentary, we document methodological problems in the Campbell Collaboration's new school-based anti-bullying interventions review, including (1) unexplained deviations from the protocol; (2) inadequate documentation of search strategies; (3) inconsistent reports on the number of included studies; (4) undocumented risk of bias ratings; (5) assessments of selective outcome reporting bias that are not transparent, not replicable, and appear to systematically underestimate risk of bias; (6) unreliable assessments of risk of publication bias; (7) use of a composite scale that conflates distinct risks of bias; and (8) failure to consider issues related to the strength of the evidence and risks of bias in interpreting results and drawing conclusions. Readers who are unaware of these problems may place more confidence in this review than is warranted. Campbell Collaboration editors declined to publish our comments and declined to issue a public statement of concern about this review. Conclusions Systematic reviews are expected to use transparent methods and follow relevant methodological standards. Readers should be concerned when these expectations are not met, because transparency and rigor enhance the trustworthiness of results and conclusions. In the tradition of Donald T. Campbell, there is need for more public debate about the methods and conclusions of systematic reviews, and greater clarity regarding applications of (and adherence to) published standards for systematic reviews.

show abstract

The methodological and reporting characteristics of Campbell reviews: A systematic review

Cited by 15 publications

References 30 publications

Risk of bias and open science practices insystematic reviews of educational effectiveness:A meta-review

Risk of bias and open science practices insystematic reviews of educational effectiveness:A meta-review

Citationchaser: A tool for transparent and efficient forward and backward citation chasing in systematic searching

The Campbell Collaboration’s systematic review of school-based anti-bullying interventions does not meet mandatory methodological standards

Contact Info

Product

Resources

About