Deep Learning Approach for Negation and Speculation Detection for Automated Important Finding Flagging and Extraction in Radiology Report: Internal Validation and Technique Comparison Study (Preprint)

Weng, Kung-Hsun; Chen, Chia‐Jung; Liu, Chung-Feng

doi:10.2196/preprints.46348

Cited by 1 publication

(1 citation statement)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Moreover, once trained for a specific task, the models lose their flexibility for other tasks, which restricts their versatility. Additionally, many BERT-based models, despite good performance, lack external validation [40][41][42][43]. Consequently, these models fall short of serving as a universal tool for labeling across various radiology reports.…”

Section: Discussionmentioning

confidence: 99%

Labeling Radiology Report With GPT-4 Prompt Engineering: Comparative Study of in-Context Prompting (Preprint)

Kim,

Shin

et al. 2024

Preprint

View full text Add to dashboard Cite

BACKGROUND Large language models, such as Generative Pre-trained Transformer-4 (GPT-4), utilize a method known as in-context learning, which enhances the model's responses by understanding the context provided within the input text. OBJECTIVE This study aims to assess the labeling efficacy of Generative Pre-trained Transformer-4 in radiology reports and to validate the performance enhancement through in-context learning. METHODS In this retrospective study, radiology reports were obtained utilizing the Medical Information Mart for Intensive Care III (MIMIC-III) database, and the reports were manually labeled by two radiologists for performance evaluation. Two experimental prompts were defined for comparison: the “Basic prompt,” which included sections for “Task” and “Output,” and the “In-context prompt,” which added a “Context” section for additional information. Labeling experiments were conducted on head CT reports for multi-label classification of ten predefined labels (mass, hemorrhage, infarct, vascular, white matter, volume loss, hydrocephalus, pneumocephalus, foreign body, and fracture) - Experiment 1. Labeling abdomen CT reports for multi-label classification of actionable findings based on four different sections (gastrointestinal, genitourinary, musculoskeletal, and vascular) - Experiment 2. Precision, recall, F1-scores, and accuracy were compared between the two prompting scenarios. RESULTS In Experiment 1, for most labels, In-context prompts demonstrated a notable improvement in F1 scores (up to 0.658) and accuracy (up to 0.155), except for hemorrhage and pneumocephalus labels. Statistically significant differences were observed in four labels (vascular, hydrocephalus, mass, foreign body). For Experiment 2, the In-context prompt significantly enhanced F1 scores (by up to 0.306) and accuracy (by up to 0.107) across all labels, compared to Basic prompts. CONCLUSIONS Our study demonstrated that Generative Pre-trained Transformer-4 with prompt engineering has commendable performance in various labeling tasks in real-world radiology reports. It offers a flexible, researcher-tailored approach to labeling tasks using in-context learning.

show abstract

Section: Discussionmentioning

confidence: 99%