Data extraction for evidence synthesis using a large language model: A proof‐of‐concept study

Gartlehner, Gerald; Kahwati, Leila; Hilscher, Rainer; Thomas, Ian; Kugley, Shannon; Crotty, Karen; Viswanathan, Meera; Nussbaumer‐Streit, Barbara; Booth, Graham; Erskine, Nathaniel; Konet, Amanda; Chew, Robert

doi:10.1002/jrsm.1710

Cited by 11 publications

(3 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…LLMs have shown high accuracy in screening relevant titles, abstracts, and full-text 22,23 , and effectively conducting quality assessment and risk-of-bias evaluation 24 . Moreover, the LLM system has been employed to automate the extraction of Population, Intervention, Comparator, and Outcome (PICO) elements 25,26 , the generation of evidence via data extraction 27,28 , and the conduction of the network meta-analyses using generated R scripts and extracted data elements 29 . However, there has been limited exploration into the holistic integration of all steps 26 , including querying relevant articles from databases like PubMed, screening abstracts and full texts based on the PICO eligibility criteria user-defined, and extracting user-specified data elements into a computable format for downstream analysis.…”

Section: Introductionmentioning

confidence: 99%

WITHDRAWN: AID-SLR: A Generative Artificial Intelligence-Driven Automated System for Systematic Literature Review

Lee,

Datta,

Paek

et al. 2024

Preprint

View full text Add to dashboard Cite

Systematic literature reviews (SLRs) are crucial for generating research evidence, supporting clinical decisions, advancing scientific knowledge, and informing policymaking. Despite their importance, manual SLRs are time-consuming, costly, and prone to errors. The increasing volume of published data and the complexity of clinical trials necessitate more efficient approaches. We present an automated SLR system using large language models (LLMs), designed to streamline the entire SLR process from initial query to data extraction, and customizable for various study fields. We developed an LLM-assisted SLR system, AID-SLR, accompanied by a user interface (UI) comprising 6 modules 1) Query, 2) Inclusion/Exclusion (I/E) criteria, 3) Abstract screening, 4) Full-text screening, 5) Data extraction, and 6) Data summary. The LLM model was utilized for abstract screening, full-text screening, and data extraction and its performance was evaluated using precision, recall, and F1 scores. We selected a non-small cell lung cancer use case to evaluate the system’s performance. We additionally compared the performance of GPT-4 and GPT-4o models, focusing on data extraction across different categories. A qualitative evaluation was conducted to assess common error types and the reliability of extracted information. AID-SLR is user-centric, allowing users to specify study criteria and provide additional information and feedback. The LLM prompts are generalizable and automatically incorporate the user-entered details and instructions on the UI, such as domain-specific guidelines, thereby enabling easy adoption of the system to different study and disease areas. AID-SLR effectively screens relevant studies and extracts data elements. The system demonstrated high precision, recall, and F1 scores in screening both Irrelevant (1, 0.9286, and 0.9630, respectively) and Relevant (0.9737, 1, and 0.9867, respectively) articles, with an overall accuracy of 98.04%. Data extraction was granular with promising performance, successfully identifying a wide range of treatment-related outcomes and statistical values. For data extraction, GPT-4o outperformed GPT-4, achieving higher precision (0.9984 vs. 0.9819), recall (0.9989 vs. 0.9519), and F1-score (0.9987 vs. 0.9651). GPT-4o also exhibited superior performance in cohort identification and value extraction, with fewer errors and more accurate capture of study design and demographic information. Our LLM system and UI provided a seamless end-to-end solution for automated SLRs. This automated SLR system can contribute to reducing the time, cost, and human errors associated with traditional manual SLRs. Integrating this model with other AI tools for comprehensive data analysis could further enhance its utility in SLRs.

show abstract

Section: Introductionmentioning

confidence: 99%

WITHDRAWN: AID-SLR: A Generative Artificial Intelligence-Driven Automated System for Systematic Literature Review

Lee,

Datta,

Paek

et al. 2024

Preprint

View full text Add to dashboard Cite

show abstract

“…10 The efficacy of LLMs for screening and data extraction for systematic reviews has been highly mixed, but there appear to be scenarios where they can help with these tasks. 11,12 We quantified the agreement of five individual LLMs with human consensus in the assessment of evidence appraisal tools of different levels of complexity: reporting (PRISMA) and methodological rigor (AMSTAR) of systematic reviews, and degree of pragmatism of clinical trials (PRECIS-2). We assessed how much complexity of assessment can be handled by current models, which one performs best, and whether the combination of multiple LLMs increases accuracy.…”

Section: Introductionmentioning

confidence: 99%

“…10 The efficacy of LLMs for screening and data extraction for systematic reviews has been highly mixed, but there appear to be scenarios where they can help with these tasks. 11,12…”

Section: Introductionmentioning

confidence: 99%

Benchmarking Human-AI Collaboration for Common Evidence Appraisal Tools

Woelfle,

Hirt,

Janiaud

et al. 2024

Preprint

View full text Add to dashboard Cite

Background: It is unknown whether large language models (LLMs) may facilitate time- and resource-intensive text-related processes in evidence appraisal. Objectives: To quantify the agreement of LLMs with human consensus in appraisal of scientific reporting (PRISMA) and methodological rigor (AMSTAR) of systematic reviews and design of clinical trials (PRECIS-2). To identify areas, where human-AI collaboration would outperform the traditional consensus process of human raters in efficiency. Design: Five LLMs (Claude-3-Opus, Claude-2, GPT-4, GPT-3.5, Mixtral-8x22B) assessed 112 systematic reviews applying the PRISMA and AMSTAR criteria, and 56 randomized controlled trials applying PRECIS-2. We quantified agreement between human consensus and (1) individual human raters; (2) individual LLMs; (3) combined LLMs approach; (4) human-AI collaboration. Ratings were marked as deferred (undecided) in case of inconsistency between combined LLMs or between the human rater and the LLM. Results: Individual human rater accuracy was 89% for PRISMA and AMSTAR, and 75% for PRECIS-2. Individual LLM accuracy was ranging from 63% (GPT-3.5) to 70% (Claude-3-Opus) for PRISMA, 53% (GPT-3.5) to 74% (Claude-3-Opus) for AMSTAR, and 38% (GPT-4) to 55% (GPT-3.5) for PRECIS-2. Combined LLM ratings led to accuracies of 75-88% for PRISMA (4-74% deferred), 74-89% for AMSTAR (6-84% deferred), and 64-79% for PRECIS-2 (18-88% deferred). Human-AI collaboration resulted in the best accuracies from 89-96% for PRISMA (25/35% deferred), 91-95% for AMSTAR (27/30% deferred), and 80-86% for PRECIS-2 (76/71% deferred). Conclusions: Current LLMs alone appraised evidence worse than humans. Human-AI collaboration may reduce workload for the second human rater for the assessment of reporting (PRISMA) and methodological rigor (AMSTAR) but not for complex tasks such as PRECIS-2.

show abstract

Performance of two large language models for data extraction in evidence synthesis

Konet,

Thomas,

Gartlehner

et al. 2024

Research Synthesis Methods

View full text Add to dashboard Cite

Accurate data extraction is a key component of evidence synthesis and critical to valid results. The advent of publicly available large language models (LLMs) has generated interest in these tools for evidence synthesis and created uncertainty about the choice of LLM. We compare the performance of two widely available LLMs (Claude 2 and GPT‐4) for extracting pre‐specified data elements from 10 published articles included in a previously completed systematic review. We use prompts and full study PDFs to compare the outputs from the browser versions of Claude 2 and GPT‐4. GPT‐4 required use of a third‐party plugin to upload and parse PDFs. Accuracy was high for Claude 2 (96.3%). The accuracy of GPT‐4 with the plug‐in was lower (68.8%); however, most of the errors were due to the plug‐in. Both LLMs correctly recognized when prespecified data elements were missing from the source PDF and generated correct information for data elements that were not reported explicitly in the articles. A secondary analysis demonstrated that, when provided selected text from the PDFs, Claude 2 and GPT‐4 accurately extracted 98.7% and 100% of the data elements, respectively. Limitations include the narrow scope of the study PDFs used, that prompt development was completed using only Claude 2, and that we cannot guarantee the open‐source articles were not used to train the LLMs. This study highlights the potential for LLMs to revolutionize data extraction but underscores the importance of accurate PDF parsing. For now, it remains essential for a human investigator to validate LLM extractions.

show abstract

Data extraction for evidence synthesis using a large language model: A proof‐of‐concept study

Cited by 11 publications

References 25 publications

WITHDRAWN: AID-SLR: A Generative Artificial Intelligence-Driven Automated System for Systematic Literature Review

WITHDRAWN: AID-SLR: A Generative Artificial Intelligence-Driven Automated System for Systematic Literature Review

Benchmarking Human-AI Collaboration for Common Evidence Appraisal Tools

Performance of two large language models for data extraction in evidence synthesis

Contact Info

Product

Resources

About