Comparison of Chest Radiograph Interpretations by Artificial Intelligence Algorithm vs Radiology Residents

Abstract

IMPORTANCE Chest radiography is the most common diagnostic imaging examination performed in emergency departments (EDs). Augmenting clinicians with automated preliminary read assistants could help expedite their workflows, improve accuracy, and reduce the cost of care. OBJECTIVE To assess the performance of artificial intelligence (AI) algorithms in realistic radiology workflows by performing an objective comparative evaluation of the preliminary reads of anteroposterior (AP) frontal chest radiographs performed by an AI algorithm and radiology residents. DESIGN, SETTING, AND PARTICIPANTS This diagnostic study included a set of 72 findings assembled by clinical experts to constitute a full-fledged preliminary read of AP frontal chest radiographs. A novel deep learning architecture was designed for an AI algorithm to estimate the findings per image. The AI algorithm was trained using a multihospital training data set of 342 126 frontal chest radiographs captured in ED and urgent care settings. The training data were labeled from their associated reports. Image-based F1 score was chosen to optimize the operating point on the receiver operating characteristics (ROC) curve so as to minimize the number of missed findings and overcalls per image read. The performance of the model was compared with that of 5 radiology residents recruited from multiple institutions in the US in an objective study in which a separate data set of 1998 AP frontal chest radiographs was drawn from a hospital source representative of realistic preliminary reads in inpatient and ED settings. A triple consensus with adjudication process was used to derive the ground truth labels for the study data set. The performance of AI algorithm and radiology residents was assessed by comparing their reads with ground truth findings. All studies were conducted through a web-based clinical study application system. The triple consensus data set

Effect of a comprehensive deep-learning model on the accuracy of chest x-ray interpretation by radiologists: a retrospective, multireader multicase study

“…The diagnostic accuracy of the model also compared favourably with that of previously published models (eg, mean ChestNet AUC 0•78). 7,8,10,22,23,44 The accuracy of the model can be at least partly attributed to the large number of cases labelled by radiologists for model training. The evaluated chest x-ray model was trained on more than 800 000 images, each labelled by radiologists using a prospectively defined ontology tree of chest x-ray findings.…”

“…The evaluated chest x-ray model was trained on more than 800 000 images, each labelled by radiologists using a prospectively defined ontology tree of chest x-ray findings. Many other largescale attempts to train deep-learning models on chest x-ray data have relied on text mining from the original radiology reports, 8,45 a process that has been criticised for inconsistency and inaccuracy. 46 Furthermore, the model uses all common chest x-ray projections (anteriorposterior, posterior-anterior, and lateral), which represents the standard of care in real-world settings.…”

“…For these reasons, several attempts have been made to create artificial intelligence (AI) systems to aid radiologists in the interpretation of chest x-rays. 7,8 Deep-learning diagnostic image processing algorithms based on convolutional neural networks have shown strong performance. 9 However, although usually highly accurate, most deeplearning systems have a narrow scope and are often limited to a single finding or a small number of findings, 7,10 restricting their use in clinical practice.…”

Performance of an artificial intelligence tool with real-time clinical workflow integration – Detection of intracranial hemorrhage and pulmonary embolism

“…Roa et al, studied more closely retrospective peer review systems to minimize false negatives in particular, whereby this AI tool could function as a real-time prospective, peer review for radiologists [19]. Earlier research evaluated on rather small data sets with a high percentage of positive cases or even exclusively positive cases, which does not represent real-time clinical workflow and might influence diagnostic accuracy [17,[27][28][29].…”

Artificial intelligence, chest radiographs, and radiology trainees: a powerful combination to enhance the future of radiologists?

“…AI and deep learning are currently being tested for imaging processing in several anatomical regions and various clinical scenarios, including disorders of the chest (7,8). In this context, we read with great interest the recently published paper by Wu et al (7), investigating the performance of AI model and human third-year radiology residents in interpreting chest radiographs.…”