Single-reviewer abstract screening missed 13 percent of relevant studies: a crowd-based, randomized controlled trial

Abstract

Objectives: To determine the accuracy of single-reviewer screening in correctly classifying abstracts as relevant or irrelevant for literature reviews. Study Design and Setting: We conducted a crowd-based, parallel-group randomized controlled trial. Using the Cochrane Crowd platform, we randomly assigned eligible participants to 100 abstracts each of a pharmacological or a public health topic. After completing a training exercise, participants screened abstracts online based on predefined inclusion and exclusion criteria. We calculated sensitivities and specificities of single-and dual-reviewer screening using two published systematic reviews as reference standards. Results: Two hundred and eighty participants made 24,942 screening decisions on 2,000 randomly selected abstracts from the reference standard reviews. On average, each abstract was screened 12 times. Overall, single-reviewer abstract screening missed 13% of relevant studies (sensitivity: 86.6%; 95% confidence interval [CI], 80.6%e91.2%). By comparison, dual-reviewer abstract screening missed 3% of relevant studies (sensitivity: 97.5%; 95% CI, 95.1%e98.8%). The corresponding specificities were 79.2% (95% CI, 77.4%e80.9%) and 68.7% (95% CI, 66.4%e71.0%), respectively. Conclusions: Single-reviewer abstract screening does not appear to fulfill the high methodological standards that decisionmakers expect from systematic reviews. It may be a viable option for rapid reviews, which deliberately lower methodological standards to provide decision makers with accelerated evidence synthesis products.

Cochrane Rapid Reviews Methods Group offers evidence-informed guidance to conduct rapid reviews

“…We also undertook a second scoping review that identified 14 empirical studies that evaluated RR methods, which we mapped to stages of review conduct [23]. The RRMG also led two methodological studies: one that assessed the impact of limiting inclusion criteria solely to English language publications [24] and one that assessed the accuracy of single-reviewer screening vs. dualreviewer screening as part of an online parallel group randomized controlled trial using the Cochrane Crowd platform [25]. Collectively, this work formed the evidentiary base for the subsequent RR methods options survey.…”

Choose your shortcuts wisely: COVID-19 rapid reviews of traditional, complementary and integrative medicine

“… 35 Trade-offs in comprehensiveness and rigour increase the risk of bias with resultant type 1 (false positive) and type 2 (false negative) errors, and reduce the certainty/quality of the review findings. 35 This concept is critical to consider in RRs and particularly for TCIM, as the conclusions of the review are rarely based on the totality of the literature, 36 but instead a limited subset, which is therefore more prone to bias.…”

Discrete Choice Experiment to Determine Preferences of Decision-Makers in Healthcare for Different Formats of Rapid Reviews.

“…If screening and data extraction are performed by one person, errors might remain undetected. For example, a recently published trial shows that single-reviewer abstract screening misses 13% of relevant studies [25]. Finally, a lack of quality assessment of the included articles may limit the validity of a rapid review as a whole [12,26].…”