Multi-Radiologist User Study for Artificial Intelligence-Guided Grading of COVID-19 Lung Disease Severity on Chest Radiographs

Abstract

Rationale and Objectives: Radiographic findings of COVID-19 pneumonia can be used for patient risk stratification; however, radiologist reporting of disease severity is inconsistent on chest radiographs (CXRs). We aimed to see if an artificial intelligence (AI) system could help improve radiologist interrater agreement. Materials and Methods:We performed a retrospective multi-radiologist user study to evaluate the impact of an AI system, the PXS score model, on the grading of categorical COVID-19 lung disease severity on 154 chest radiographs into four ordinal grades (normal/minimal, mild, moderate, and severe). Four radiologists (two thoracic and two emergency radiologists) independently interpreted 154 CXRs from 154 unique patients with COVID-19 hospitalized at a large academic center, before and after using the AI system (median washout time interval was 16 days). Three different thoracic radiologists assessed the same 154 CXRs using an updated version of the AI system trained on more imaging data. Radiologist interrater agreement was evaluated using Cohen and Fleiss kappa where appropriate. The lung disease severity categories were associated with clinical outcomes using a previously published outcomes dataset using Fisher's exact test and Chi-square test for trend.Results: Use of the AI system improved radiologist interrater agreement (Fleiss k = 0.40 to 0.66, before and after use of the system). The Fleiss k for three radiologists using the updated AI system was 0.74. Severity categories were significantly associated with subsequent intubation or death within 3 days. Conclusion:An AI system used at the time of CXR study interpretation can improve the interrater agreement of radiologists.

The Applications of Artificial Intelligence in Chest Imaging of COVID-19 Patients: A Literature Review

“…However, there is no standardized method in reporting CXR findings in terms of disease severity. Li et al used the pulmonary x-ray severity (PXS) score, a DL-based algorithm providing quantitative measures of COVID-19 severity on CXR, as an adjuvant tool to radiologists' work-which, however, always decided on the severity grading and definitive radiological report-, and noticed an improvement in the assessment of the severity on a 4-point scale (normal/minimal, mild, moderate, severe) and in the inter-reader agreement, with no need for radiologists' training on the use of the score [59,60]. Li et al also found that the severity scores were significantly associated with intubation/death within 3 days from the admission, in CXR rated moderate or severe [59].…”

“…Li et al used the pulmonary x-ray severity (PXS) score, a DL-based algorithm providing quantitative measures of COVID-19 severity on CXR, as an adjuvant tool to radiologists' work-which, however, always decided on the severity grading and definitive radiological report-, and noticed an improvement in the assessment of the severity on a 4-point scale (normal/minimal, mild, moderate, severe) and in the inter-reader agreement, with no need for radiologists' training on the use of the score [59,60]. Li et al also found that the severity scores were significantly associated with intubation/death within 3 days from the admission, in CXR rated moderate or severe [59]. Mushtaq et al reported in their retrospective study that an AI-powered severity score based on the percentage of pixels involved by opacity or consolidation for each lung at the CXR, adjusted at the multivariate analysis for demographics and comorbidities, showed that a value ≥30 at the hospital admission CXR was an independent predictor for mortality and ICU admission for COVID19 (p < 0.001), and found a significant link with admission pO2/FiO2 levels [61].…”

Diagnostic Test Accuracy of Deep Learning Detection of COVID-19: A Systematic Review and Meta-Analysis

“…Recently, Li et al (11) carried out a multi-reader study for the grading of COVID-19 in chest radiography and observed that the AI system improved radiologist performance. Since the deep learning technique has been found useful in the diagnosis of COVID-19 (12,13), combining radiologist interpretation with the DL approach gives a promising result for the detection of COVID-19 (13).…”

Radiology Implementation Considerations for Artificial Intelligence (AI) Applied to COVID-19, From the AJR Special Series on AI Applications

“…For example, multiple groups have developed AI tools to grade lung disease severity on chest radiographs [28][29][30]; AI-determined severity scores have been correlated with clinical outcomes such as intubation and death [30]. Such AI tools have also been used to improve the interrater agreement of radiologists evaluating COVID-19 lung disease severity on chest radiographs [31].…”