Combining abbreviated literature searches with single-reviewer screening: three case studies of rapid reviews

Abstract

Background Decision-makers increasingly request rapid answers to clinical or public health questions. To save time, personnel, and financial resources, rapid reviews streamline the methodological steps of the systematic review process. We aimed to explore the validity of a rapid review approach that combines a substantially abbreviated literature search with single-reviewer screening of abstracts and full texts using three case studies. Methods We used a convenience sample of three ongoing Cochrane reviews as reference standards. Two reviews addressed oncological topics and one addressed a public health topic. For each of the three topics, three reviewers screened the literature independently. Our primary outcome was the change in conclusions between the rapid reviews and the respective Cochrane reviews. In case the rapid approach missed studies, we recalculated the meta-analyses for the main outcomes and asked Cochrane review authors if the new body of evidence would change their original conclusion compared with the reference standards. Additionally, we assessed the sensitivity of the rapid review approach compared with the results of the original Cochrane reviews. Results For the two oncological topics (case studies 1 and 2), the three rapid reviews each yielded the same conclusions as the Cochrane reviews. However, the authors would have had less certainty about their conclusion in case study 2. For case study 3, the public health topic, only one of the three rapid reviews led to the same conclusion as the Cochrane review. The other two rapid reviews provided insufficient information for the authors to draw conclusions. Using the rapid review approach, the sensitivity was 100% (3 of 3) for case study 1. For case study 2, the three rapid reviews identified 40% (4 of 10), 50% (5 of 10), and 60% (6 of 10) of the included studies, respectively; for case study 3, the respective numbers were 38% (8 of 21), 43% (9 of 21), and 48% (10 of 21). Conclusions Within the limitations of these case studies, a rapid review approach that combines abbreviated literature searches with single-reviewer screening may be feasible for focused clinical questions. For complex public health topics, sensitivity seems to be insufficient.

Assessing the Accuracy of Machine-Assisted Abstract Screening With DistillerAI: A User Study. Methods Research Report

“…In a case study, this approach reduced the number of abstracts that needed to be screened by up to 90 percent without missing studies that would have changed conclusions. 20 It is conceivable that such a targeted literature search approach could improve the performance of semi-automated screening tools because they would have to deal with less noise.…”

“…18 Natural language processing technology: A semantic technology process that is used to normalize variants (e.g., different conjugations of a verb) of a single concept, and to identify complex concepts (e.g., terms made up of several words) in a text. 19,20 Prediction: A forecast of whether a record is relevant (include) or irrelevant (exclude) for a given systematic review. Semi-automated screening/text mining tool: Any Web-based application that employs a combination of text mining and text classification to assist systematic reviewers to make decisions during the title and abstract screening process.…”

Methodological guidance for rapid reviews in healthcare: A scoping review

“…71 Although consensus exists that methodological shortcuts in the conduct of a review can lead to bias, retrospective comparisons of systematic reviews and RRs indicate that both types of reviews apparently lead to comparable conclusions. Affengruber et al 72 found that, when compared to Cochrane reviews, a RR approach that combines abbreviated literature searches and singlereviewer abstract screening had low accuracy. However, this seems to depend on the research area.…”

“…The RR approach would have led to comparable conclusions as the original Cochrane review for oncological topics, but not for public health topics. 72 Furthermore, the varying quality of systematic reviews has to be taken into account, hampering the identification of marked differences between the RRs and systematic reviews in general. 73 For example, in an analysis of the methodological restrictions of RRs compared to systematic reviews, a duplicate study selection and data extraction was only performed in 37% of reviews labeled as "systematic."…”

“…However, this seems to depend on the research area. The RR approach would have led to comparable conclusions as the original Cochrane review for oncological topics, but not for public health topics 72 …”