Machine learning in systematic reviews: Comparing automated text clustering with Lingo3G and human researcher categorization in a rapid review

Abstract

Systematic reviews are resource‐intensive. The machine learning tools being developed mostly focus on the study identification process, but tools to assist in analysis and categorization are also needed. One possibility is to use unsupervised automatic text clustering, in which each study is automatically assigned to one or more meaningful clusters. Our main aim was to assess the usefulness of an automated clustering method, Lingo3G, in categorizing studies in a simplified rapid review, then compare performance (precision and recall) of this method compared to manual categorization. We randomly assigned all 128 studies in a review to be coded by a human researcher blinded to cluster assignment (mimicking two independent researchers) or by a human researcher non‐blinded to cluster assignment (mimicking one researcher checking another's work). We compared time use, precision and recall of manual categorization versus automated clustering. Automated clustering and manual categorization organized studies by population and intervention/context. Automated clustering failed to identify two manually identified categories but identified one additional category not identified by the human researcher. We estimate that automated clustering has similar precision to both blinded and non‐blinded researchers (e.g., 88% vs. 89%), but higher recall (e.g., 89% vs. 84%). Manual categorization required 49% more time than automated clustering. Using a specific clustering algorithm, automated clustering can be helpful with categorization of and identifying patterns across studies in simpler systematic reviews. We found that the clustering was sensitive enough to group studies according to linguistic differences that often corresponded to the manual categories.

The effect of machine learning tools for evidence synthesis on resource use and time-to-completion: protocol for a retrospective pilot study

“…Automatic classification and exclusion of non-randomized designs with a study design classifier saved Cochrane Crowd from manually screening more than 40% of identified references in 2018 [25]. We have also reported that categorizing studies using automated clustering used 33% of the time compared to manual categorization [26].…”

Automation of systematic reviews of biomedical literature: a systematic review of studies indexed in PubMed

“…In the third most used strategy, review updates, all included papers and excluded records of a published review are used for training, and the aim is to predict the inclusion of a record from new search results in the updated review (N = 12/89, 13.5%) 34,46,81,83,89,93,112,124,125,129,152,153 . The priority ranking strategy (N = 10/89, 11.2%) 33,36,38,65,66,75,91,132,134,139 was used least often. This strategy predicts the priority of records after single training round.…”

“…The off-the shelf or freeware screening automation software were Abstrackr 77,90,107,108,112,114,131 , EPPI Reviewer 50,114,139,147 , RobotAnalyst 80,87, 90,112 , Distiller SR 90,109,132 , Rayyan 87,120 , Systematic Review Accelerator 18,119 , RCT Tagger 141,143 , SWIFT Review 91,132 , SyRF, 84,89 , ASR (Automated Systematic Review) 95 , ASReview 146 , Aggregator 52 , ATCER 79 , Cochrane RCT Classi er 126 , Covidence 87 , Curious Snake 41 , DoCTER 86 , GAP Screener 35 , MetaPreg 130 , Research Screener 118 , revtools 100 , RobotAnalyst, and TeMMPo 75 . The detailed description of these tools is beyond the scope of this study.…”

Conducting a systematic review in six weeks – experiences with and evaluation of an intensive team pilot project

“…At the Norwegian Institute of Public Health (NIPH) a key focus has been on how to reduce the lead time for our commissions without compromising methodological quality. To achieve this goal, a lean-inspired (1) project to improve work ows was conducted, and a machine learning team was founded to evaluate tools to speed up the review process (2,3,4,5).…”