Machine Learning Model for Chest Radiographs: Using Local Data to Enhance Performance

Abstract

Purpose To develop and assess the performance of a machine learning model which screens chest radiographs for 14 labels, and to determine whether fine-tuning the model on local data improves its performance. Generalizability at different institutions has been an obstacle to machine learning model implementation. We hypothesized that the performance of a model trained on an open-source dataset will improve at our local institution after being fine-tuned on local data. Methods In this retrospective, institutional review board approved study, an ensemble of neural networks was trained on open-source datasets of chest radiographs for the detection of 14 labels. This model was then fine-tuned using 4510 local radiograph studies, using radiologists’ reports as the gold standard to evaluate model performance. Both the open-source and fine-tuned models’ accuracy were tested on 802 local radiographs. Receiver-operator characteristic curves were calculated, and statistical analysis was completed using DeLong’s method and Wilcoxon signed-rank test. Results The fine-tuned model identified 12 of 14 pathology labels with area under the curves greater than .75. After fine-tuning with local data, the model performed statistically significantly better overall, and specifically in detecting six pathology labels ( P < .01). Conclusions A machine learning model able to accurately detect 14 labels simultaneously on chest radiographs was developed using open-source data, and its performance was improved after fine-tuning on local site data. This simple method of fine-tuning existing models on local data could improve the generalizability of existing models across different institutions to further improve their local performance.

Integration of artificial intelligence in clinical laboratory medicine: Advancements and challenges

“…A notable illustration of this approach is found in the work of Mohn et al, who enhanced the adaptability of existing models across diverse institutions by fine-tuning them using locally sourced data. 117 Ethical concerns, encompassing matters of accountability, privacy, and transparency, pose limitations on the widespread adoption of AI. It is imperative to bolster the framework for responsibility management and delineate the roles and obligations of medical institutions, developers, and healthcare professionals at all AI implementation stages.…”

“…Therefore, prior to implementing AI‐based analysis, it is imperative to develop dedicated server programs and establish standardized data formats. A notable illustration of this approach is found in the work of Mohn et al., who enhanced the adaptability of existing models across diverse institutions by fine‐tuning them using locally sourced data 117 …”

Editor’s Corner: August 2023

“…In addition, this study addresses the generalizability issue in implementing machine learning models in medical imaging. 2…”