SEETrials: Leveraging Large Language Models for Safety and Efficacy Extraction in Oncology Clinical Trials

Abstract

Objective Conference abstracts provide preliminary evidence for clinical trial outcomes. This study aims to develop an automatic extraction system to precisely extract and convert granular safety and efficacy information from abstracts into a computable format for timely downstream analysis. Materials and Methods We collected multiple myeloma clinical trial abstracts from ASCO, ASH, and PubMed (2012-2023) to develop SEETrials, a GPT-4 based system. Qualitative and quantitative evaluations were conducted. Descriptive data analysis on efficacy and safety entities was performed. The generalizability of the system was tested in three other cancer trial studies. Results SEETrials achieved precision, recall, and F1 of 95.5%, 93.9%, and 94.7% across 70 data elements. Analysis of 245 multiple myeloma clinical trial abstracts revealed variations in safety and efficacy entity distribution across different modalities and phases. Application to other cancer trial studies demonstrated consistent performance with precision, recall, and F1 of 96.9%, 95.4%, and 96.1%, respectively. Discussion Qualitative error analysis identified a common source of inaccuracy, notably in cohort identification and categorization. Phase1/2 studies prioritize early indications of treatment efficacy, while phase 2/3 studies emphasize prolonged effects. Adverse event distribution is consistent across phases, with noticeable increases in fatal events in later phases, suggesting longer follow-up or more patients may reveal severe adverse events missed in phase 1 safety studies. Conclusion SEETrials displayed high accuracy and generalizability to diverse drug modalities and disease domains. Its capacity to streamline large-scale dataset analysis is crucial for advancing clinical trial research, ensuring timely and accurate data extraction, and facilitating efficient dissemination.

WITHDRAWN: AID-SLR: A Generative Artificial Intelligence-Driven Automated System for Systematic Literature Review

“…We acknowledged some limitations in our study. Our whole system was tested using one use case, although a similar LLM-based data extraction module focused on abstracts was previously tested in other indications and contexts 28 . Evaluating the system’s performance across different disease areas and various types of clinical trials or real-world studies is crucial for establishing its generalizability and robustness.…”

“…LLMs have shown high accuracy in screening relevant titles, abstracts, and full-text 22,23 , and effectively conducting quality assessment and risk-of-bias evaluation 24 . Moreover, the LLM system has been employed to automate the extraction of Population, Intervention, Comparator, and Outcome (PICO) elements 25,26 , the generation of evidence via data extraction 27,28 , and the conduction of the network meta-analyses using generated R scripts and extracted data elements 29 . However, there has been limited exploration into the holistic integration of all steps 26 , including querying relevant articles from databases like PubMed, screening abstracts and full texts based on the PICO eligibility criteria user-defined, and extracting user-specified data elements into a computable format for downstream analysis.…”