Automation of systematic reviews of biomedical literature: a systematic review of studies indexed in PubMed

Abstract

Background The demand for high quality systematic literature reviews (SLRs) is growing for evidence-based medical decision making. SLRs are costly and require the scarce resource of highly skilled reviewers. Automation technology has been proposed to save workload and expedite the SLR workflow. Objectives We aimed to provide a comprehensive overview of SLR automation studies indexed in PubMed, focusing on the applicability of these technologies in real world practice. Methods In November 2022, we ran a combined search syntax of four published SLRs on SLR automation. Full-text English peer-reviewed articles were included if they reported Studies on SLR Automation Methods (SSAM), or Automated SLRs (ASLR). Bibliographic analyses and knowledge-discovery studies were excluded. Record screening was performed by single reviewers, the selection of full text papers was performed in duplicate. We summarized the publication details, automated review stages, automation goals, applied tools, data sources, methods, results and Google Scholar citations of SLR automation studies. Results From 5321 records screened by title and abstract, we included 123 full text articles, out of which 108 were SSAMs and 15 ASLRs. Automation was applied for search, record screening, full-text selection, data extraction, risk of bias assessment, evidence synthesis, assessment of evidence quality and reporting in 19 (15.4%), 89 (72.4%), 6 (4.9%), 13 (10.6%), 9 (7.3%), 2 (1.6%), 2 (1.6%), and 2 (1.6%) studies, respectively. Multiple SLR stages were automated by 11 (8.9%) studies. The performance of automated record screening varied largely across SLR topics. In published ASLRs we found examples of automated search, record screening, full-text selection and data extraction. In some ASLRs automation complemented fully manual reviews to increase sensitivity rather than to save workload. Reporting of automation details were often incomplete in ASLRs. Conclusions Automation techniques are being developed for all SLRs stages, but with limited real-world adoption. Most SLR automation tools target single SLR stages, with modest time savings for the entire SLR process and varying sensitivity and specificity across studies. Therefore, the real-world benefits of SLR automation remain uncertain. Standardizing the terminology, reporting, and metrics of study reports could enhance the adoption of SLR automation techniques in real-world practice.

Artificial Intelligence in Medicine: A Systematic Review of Guidelines on Reporting and Interpreting Studies

“…This variety makes it di cult for authors or clinicians to get a rm grip on what represents best practice in reporting, and how detailed a standardized medical AI study report should be. This may delay the clinical uptake of medical AI technologies, hamper evidence syntheses and consequently impede reliable information retrieval, which is essential for the development of automated systematic literature reviews(53). AI is concerned about clinical implementation pilot studies of AI-based technologies.…”