Diagnostic accuracy and inter-observer agreement with the CO-RADS lexicon for CT chest reporting in COVID-19

Abstract

Purpose To measure the diagnostic accuracy and inter-observer agreement with the use of COVID-19 Reporting and Data System (CO-RADS) for detection of COVID-19 on CT chest imaging. Methods This retrospective study included 164 consecutive patients with clinical suspicion of COVID-19 in whom a CT chest examination was performed at a single institution between April 2020 and July 2020. Of them, 101 patients was RT-PCR positive for COVID-19. Six readers with varying radiological experience (two each of chest radiologists, general radiologists, and radiologists in training) independently assigned a CO-RADS assessment category for each CT chest study. The Fleiss’ K was used to quantify inter-observer agreement. The inter-observer agreement was also assessed based on the duration of onset of symptoms to CT scan. ROC curve analysis was used to determine the diagnostic accuracy of CO-RADS. The area under curve was calculated to determine the reader accuracy for detection of COVID-19 lung involvement with RT-PCR as reference standards. The data sets were plotted in ROC space, and Youden’s J statistic was calculated to determine the threshold cut-off CO-RADS category for COVID-19 positivity. Results There was overall moderate inter-observer agreement between all readers (Fleiss’ K 0.54 [95% CI 0.54, 0.54]), with substantial agreement among chest radiologists (Fleiss’ K 0.68 [95% CI 0.67, 0.68]), general radiologists (Fleiss’ K 0.61 [95% CI 0.61, 0.61]), and moderate agreement among radiologists-in-training (Fleiss’ K 0.56 [95% CI 0.56, 0.56]). There was overall moderate inter-observer agreement in early disease (stages 1 and 2), with cumulative Fleiss’ K 0.45 [95% CI 0.45, 0.45]). The overall AUC for CO-RADS lexicon scheme to accurately diagnose COVID-19 yielded 0.92 (95% CI 0.91, 0.94) with strong concordance within and between groups, of chests radiologists with AUC of 0.91 (95% CI 0.88, 0.94), general radiologists with AUC 0.96 (95% CI 0.94, 0.98), and radiologists in training with AUC of 0.90 (95% CI 0.87, 0.94). For detecting COVID-19, ROC curve analysis yielded CO-RADS > 3 as the cut-off threshold with sensitivity 90% (95% CI 0.88, 0.93), and specificity of 87% (95% CI 0.83, 0.91). Conclusion Readers across different levels of experience could accurately identify COVID-19 positive patients using the CO-RADS lexicon with moderate inter-observer agreement and high diagnostic accuracy.

Chest computed tomography of suspected COVID-19 pneumonia in the Emergency Department: comparative analysis between patients with different vaccination status

“…The overall agreement between the 3 readers was excellent (quadratic weighted κ > 0.8) for both CO-RADS and ACR classifications, which was concordant with recent literature [30]. Some differences in CO-RADS 3 or 4 rates for Reader 1, in CO-RADS 5 rates for Reader 2, or CO-RADS 2 rates for reader 3 (Table 3) could be considered as outliers because they were not confirmed by the other 2 readers, and they could be explained by the lower Fleiss' κ values for some categories [31,32]. The decision to consider CO-RADS 3 to 5 as COVID-19 pneumonia was due to the high PPV (70%) previously documented in symptomatic individuals [33].…”

A Systematic Review of Chest Imaging Findings in Long COVID Patients

“…Using either a computer-based or a human-based method to determine the findings affects the precision of the imaging reports. In human-based reports, the expertise and experience of the interpreter are important, an importance which is likely to be increased in the context of a new and emerging condition [ 47 , 48 ]. Although one of our aims was to evaluate what the imaging interpretations method is, only three studies [ 23 , 24 , 26 ] mentioned this point.…”

Diagnostic performance of standardized typical CT findings for COVID-19: a systematic review and meta-analysis

“…The results of a sensitivity analysis for 28 studies, including more than 50 subjects among both COVID-19 cases and non-COVID-19 cases, each, were similar to those of the primary analysis [ 9 , 20 – 24 , 26 – 28 , 30 , 33 – 38 , 41 , 45 – 48 , 50 – 52 , 54 – 57 ] (Additional file 1 : Fig. 4).…”

“…The mean or median age of patients ranged from 40 to 76 years old. There were 18 studies from developing countries (4 studies from lower-middle-income and 14 studies from upper-middle-income countries) [ 18 , 19 , 22 , 23 , 30 – 32 , 35 , 37 , 38 , 43 – 45 , 47 , 48 , 50 , 51 , 58 ] and 24 studies from developed countries (high income) [ 9 , 20 , 21 , 24 – 29 , 33 , 34 , 36 , 39 – 42 , 46 , 49 , 52 – 57 ]. The included studies were predominantly from Europe (24 studies) [ 9 , 20 , 21 , 23 – 27 , 29 , 31 , 32 , 34 , 35 , 37 , 38 , 41 , 42 , 47 – 49 , 52 , 55 – 57 ], followed by the Americas (10 studies), Asia (seven studies), and Africa (one study).…”