Labeling Noncontrast Head CT Reports for Common Findings Using Natural Language Processing

Iorga, Michael; Drakopoulos, Michael; Naidech, Andrew M.; Katsaggelos, Aggelos K.; Parrish, Todd B.; Hill, Virginia

doi:10.3174/ajnr.a7500

Cited by 3 publications

(1 citation statement)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The annotation rules for each label were clearly defined, referring to the annotation rules from the cited study [22]. Furthermore, incorporating insights from Lorga's study, labels frequently identified in noncontrast head CT scans, such as hemorrhage, fracture, and pneumocephalus, were also included [23].…”

Section: ) Experimentsmentioning

confidence: 99%

Labeling Radiology Report With GPT-4 Prompt Engineering: Comparative Study of in-Context Prompting (Preprint)

Kim,

Shin

et al. 2024

Preprint

View full text Add to dashboard Cite

BACKGROUND Large language models, such as Generative Pre-trained Transformer-4 (GPT-4), utilize a method known as in-context learning, which enhances the model's responses by understanding the context provided within the input text. OBJECTIVE This study aims to assess the labeling efficacy of Generative Pre-trained Transformer-4 in radiology reports and to validate the performance enhancement through in-context learning. METHODS In this retrospective study, radiology reports were obtained utilizing the Medical Information Mart for Intensive Care III (MIMIC-III) database, and the reports were manually labeled by two radiologists for performance evaluation. Two experimental prompts were defined for comparison: the “Basic prompt,” which included sections for “Task” and “Output,” and the “In-context prompt,” which added a “Context” section for additional information. Labeling experiments were conducted on head CT reports for multi-label classification of ten predefined labels (mass, hemorrhage, infarct, vascular, white matter, volume loss, hydrocephalus, pneumocephalus, foreign body, and fracture) - Experiment 1. Labeling abdomen CT reports for multi-label classification of actionable findings based on four different sections (gastrointestinal, genitourinary, musculoskeletal, and vascular) - Experiment 2. Precision, recall, F1-scores, and accuracy were compared between the two prompting scenarios. RESULTS In Experiment 1, for most labels, In-context prompts demonstrated a notable improvement in F1 scores (up to 0.658) and accuracy (up to 0.155), except for hemorrhage and pneumocephalus labels. Statistically significant differences were observed in four labels (vascular, hydrocephalus, mass, foreign body). For Experiment 2, the In-context prompt significantly enhanced F1 scores (by up to 0.306) and accuracy (by up to 0.107) across all labels, compared to Basic prompts. CONCLUSIONS Our study demonstrated that Generative Pre-trained Transformer-4 with prompt engineering has commendable performance in various labeling tasks in real-world radiology reports. It offers a flexible, researcher-tailored approach to labeling tasks using in-context learning.

show abstract

Section: ) Experimentsmentioning

confidence: 99%