A real‐world evaluation of the implementation of NLP technology in abstract screening of a systematic review

Abstract

The laborious and time‐consuming nature of systematic review production hinders the dissemination of up‐to‐date evidence synthesis. Well‐performing natural language processing (NLP) tools for systematic reviews have been developed, showing promise to improve efficiency. However, the feasibility and value of these technologies have not been comprehensively demonstrated in a real‐world review. We developed an NLP‐assisted abstract screening tool that provides text inclusion recommendations, keyword highlights, and visual context cues. We evaluated this tool in a living systematic review on SARS‐CoV‐2 seroprevalence, conducting a quality improvement assessment of screening with and without the tool. We evaluated changes to abstract screening speed, screening accuracy, characteristics of included texts, and user satisfaction. The tool improved efficiency, reducing screening time per abstract by 45.9% and decreasing inter‐reviewer conflict rates. The tool conserved precision of article inclusion (positive predictive value; 0.92 with tool vs. 0.88 without) and recall (sensitivity; 0.90 vs. 0.81). The summary statistics of included studies were similar with and without the tool. Users were satisfied with the tool (mean satisfaction score of 4.2/5). We evaluated an abstract screening process where one human reviewer was replaced with the tool's votes, finding that this maintained recall (0.92 one‐person, one‐tool vs. 0.90 two tool‐assisted humans) and precision (0.91 vs. 0.92) while reducing screening time by 70%. Implementing an NLP tool in this living systematic review improved efficiency, maintained accuracy, and was well‐received by researchers, demonstrating the real‐world effectiveness of NLP in expediting evidence synthesis.

Performance of a Large Language Model in Screening Citations

“…Our study further found that LLM-assisted citation screening helped save time in the systematic review process, with a more than 10-fold reduction in the time required to complete the process. Although this finding is consistent with other reports showing the advantage of citation screening using semiautomated screening software, 9 , 10 , 26 eliminating the necessity of inputting key studies would save additional time using LLM-assisted citation screening.…”

“…Second, LLM-assisted citation may have a higher accuracy than the semiautomated tool. Previous studies on semiautomated citation screening tools reported sensitivity ranging from 0.75 to 0.90, which is comparable with the accuracy of our study; however, our previous research on this tool showed a variable sensitivity of 0.24 to 0.80 for the same dataset used in the present study. Moreover, we found a higher sensitivity of 0.53 to 0.95 with lower variability in the secondary analysis, suggesting the potential advantage of LLM-assisted citation screening for discriminating the relevant literature.…”

“…First, the LLM-assisted citation screening may lead to improved efficiency and workload reduction during the screening process because although semiautomated citation screening tools using machine learning showed enhanced efficiency and workload reduction, their application requires training data for the citation screening process, inputting predefined key articles, and some processes of human reviewers. 4 , 9 , 26 In contrast, LLM-assisted citation screening does not require any further training data or efforts of human reviewers in the screening process. Our study further found that LLM-assisted citation screening helped save time in the systematic review process, with a more than 10-fold reduction in the time required to complete the process.…”

“…Our findings indicated the potential of LLM-assisted citation screening, which has substantial advantages over previously reported semiautomated screening tools. First, the LLM-assisted citation screening may lead to improved efficiency and workload reduction during the screening process because although semiautomated citation screening tools using machine learning showed enhanced efficiency and workload reduction, their application requires training data for the citation screening process, inputting predefined key articles, and some processes of human reviewers . In contrast, LLM-assisted citation screening does not require any further training data or efforts of human reviewers in the screening process.…”

Diagnostic performance of artificial intelligence tools for article screening during literature review: A systematic review (Preprint)

“…Though the AI-assisted methods developed show promise in studies among previously existing SR and meta-analysis, few have assessed their performance in real-world settings. In a study by Perlmann-Arrow et al, 2022, they developed an NLP-assisted abstract screening tool and evaluated it against a living SR on SARS-CoV2 seroprevalence being updated by investigators [7]. Results showed that the time to complete screening was reduced by 45.9% with a recall of 90%, and a mean user satisfaction rating of 4.2/5.0 when the tool was compared to the two-reviewer approach.…”