The use of artificial intelligence for automating or semi-automating biomedical literature analyses: A scoping review

Santos, Álisson Oliveira dos; Silva, Eduardo Sérgio da; Couto, Letícia Machado; Reis, Gustavo Valadares Labanca; Belo, Viní­cius Silva

doi:10.1016/j.jbi.2023.104389

Cited by 14 publications

(7 citation statements)

References 252 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…While current fully automated systems cannot replace humans in title and abstract screening, they may nevertheless be helpful. Such systems are already being used in systematic reviews and most likely their usage will continue to grow [ 57 ].…”

Section: Discussionmentioning

confidence: 99%

Title and abstract screening for literature reviews using large language models: an exploratory study in the biomedical domain

Dennstädt,

Zink,

Putora

et al. 2024

Syst Rev

View full text Add to dashboard Cite

Background Systematically screening published literature to determine the relevant publications to synthesize in a review is a time-consuming and difficult task. Large language models (LLMs) are an emerging technology with promising capabilities for the automation of language-related tasks that may be useful for such a purpose. Methods LLMs were used as part of an automated system to evaluate the relevance of publications to a certain topic based on defined criteria and based on the title and abstract of each publication. A Python script was created to generate structured prompts consisting of text strings for instruction, title, abstract, and relevant criteria to be provided to an LLM. The relevance of a publication was evaluated by the LLM on a Likert scale (low relevance to high relevance). By specifying a threshold, different classifiers for inclusion/exclusion of publications could then be defined. The approach was used with four different openly available LLMs on ten published data sets of biomedical literature reviews and on a newly human-created data set for a hypothetical new systematic literature review. Results The performance of the classifiers varied depending on the LLM being used and on the data set analyzed. Regarding sensitivity/specificity, the classifiers yielded 94.48%/31.78% for the FlanT5 model, 97.58%/19.12% for the OpenHermes-NeuralChat model, 81.93%/75.19% for the Mixtral model and 97.58%/38.34% for the Platypus 2 model on the ten published data sets. The same classifiers yielded 100% sensitivity at a specificity of 12.58%, 4.54%, 62.47%, and 24.74% on the newly created data set. Changing the standard settings of the approach (minor adaption of instruction prompt and/or changing the range of the Likert scale from 1–5 to 1–10) had a considerable impact on the performance. Conclusions LLMs can be used to evaluate the relevance of scientific publications to a certain review topic and classifiers based on such an approach show some promising results. To date, little is known about how well such systems would perform if used prospectively when conducting systematic literature reviews and what further implications this might have. However, it is likely that in the future researchers will increasingly use LLMs for evaluating and classifying scientific publications.

show abstract

Section: Discussionmentioning

confidence: 99%

Title and abstract screening for literature reviews using large language models: an exploratory study in the biomedical domain

Dennstädt,

Zink,

Putora

et al. 2024

Syst Rev

View full text Add to dashboard Cite

show abstract

“…Based on a scoping review of 273 publications on the use of AI for automating or semi-automating biomedical literature analyses, Santos et al (2023) found applications to assembly of scientific evidence, mining the biomedical literature, and quality analysis. Most studies addressed the preparation of systematic reviews rather than the development of guidelines or evidence syntheses.…”

Section: Artificial Intelligence Technologies Employedmentioning

confidence: 99%

Reviews and Reviewing: Approaches to Research Synthesis. An Annual Review of Information Science and Technology (ARIST) paper

Smith

2023

Asso for Info Science & Tech

View full text Add to dashboard Cite

Reviews have long been recognized as among the most important forms of scientific communication. The rapid growth of the primary literature has further increased the need for reviews to distill and interpret the literature. This review on Reviews and Reviewing: Approaches to Research Synthesis encompasses the evolution of the review literature, taxonomy of review literature, uses and users of reviews, the process of preparing reviews, assessment of review quality and impact, the impact of information technology on the preparation of reviews, and research opportunities for information science related to reviews and reviewing. In addition to providing a synthesis of prior research, this review seeks to identify gaps in the published research and to suggest possible future research directions.

show abstract

“…The use of automated software tools to assist in reviewing research papers has become a topic of increasing interest. Most such tools have used natural language processing (NLP) and machine learning (ML) algorithms primarily to screen the titles and abstracts of publications to determine whether they meet the search criteria for a systematic review (2)(3)(4)(5)(6)(7)(8). Several studies have also described the potential for using the representational language model BERT (Bidirectional Encoder Representations from Transformers) and the Generative Pre-trained Transformer (GPT) large language models (LLMs) for reviewing the full text of published studies (9)(10)(11)(12).…”

Section: Introductionmentioning

confidence: 99%

GPT-4 Performance on Querying Scientific Publications: Reproducibility, Accuracy, and Impact of an Instruction Sheet

Tao,

Osman,

Tzou

et al. 2024

Preprint

View full text Add to dashboard Cite

Background: Large language models (LLMs) that could efficiently screen and identify studies fulfilling specific criteria, as well as those capable of data extraction from publications, would streamline literature reviews and enhance knowledge discovery by lessening the burden on human reviewers. Methods: We created an automated pipeline utilizing OpenAI GPT-4 32K API version “2023-05-15” to evaluate the accuracy of the LLM GPT-4 when responding to queries about published papers on HIV drug resistance (HIVDR) with and without an instruction sheet. The instruction sheet contained specialized knowledge designed to assist a person trying to answer questions about an HIVDR paper. We designed 60 questions pertaining to HIVDR and created markdown versions of 60 published HIVDR papers in PubMed. We presented the 60 papers to GPT-4 in four configurations: (1) all 60 questions simultaneously; (2) all 60 questions simultaneously with the instruction sheet; (3) each of the 60 questions individually; and (4) each of the 60 questions individually with the instruction sheet. Results: GPT-4 achieved a mean accuracy of 86.9% – 24.0% higher than when the answers to papers were permuted. The overall recall and precision were 72.5% and 87.4%, respectively. The standard deviation of three replicates for the 60 questions ranged from 0 to 5.3% with a median of 1.2%. The instruction sheet did not significantly increase GPT-4’s accuracy, recall, or precision. GPT-4 was more likely to provide false positive answers when the 60 questions were submitted individually compared to when they were submitted together. Conclusions: GPT-4 reproducibly answered 3600 questions about 60 papers on HIVDR with moderately high accuracy, recall, and precision. The inability of the instruction sheet to increase these metrics suggests that more sophisticated prompt engineering approaches or the finetuning of an open source model are required to further improve the ability of an LLM to answer questions about highly specialized HIV drug resistance papers.

show abstract

The use of artificial intelligence for automating or semi-automating biomedical literature analyses: A scoping review

Cited by 14 publications

References 252 publications

Title and abstract screening for literature reviews using large language models: an exploratory study in the biomedical domain

Title and abstract screening for literature reviews using large language models: an exploratory study in the biomedical domain

Reviews and Reviewing: Approaches to Research Synthesis. An Annual Review of Information Science and Technology (ARIST) paper

GPT-4 Performance on Querying Scientific Publications: Reproducibility, Accuracy, and Impact of an Instruction Sheet

Contact Info

Product

Resources

About