Interobserver Variation in Interpreting Chest Radiographs for the Diagnosis of Acute Respiratory Distress Syndrome

Abstract

To measure the reliability of chest radiographic diagnosis of acute respiratory distress syndrome (ARDS) we conducted an observer agreement study in which two of eight intensivists and a radiologist, blinded to one another's interpretation, reviewed 778 radiographs from 99 critically ill patients. One intensivist and a radiologist participated in pilot training. Raters made a global rating of the presence of ARDS on the basis of diffuse bilateral infiltrates. We assessed interobserver agreement in a pairwise fashion. For rater pairings in which one rater had not participated in the consensus process we found moderate levels of raw (0.68 to 0.80), chance-corrected (kappa 0.38 to 0.55), and chance-independent (Phi 0. 53 to 0.75) agreement. The pair of raters who participated in consensus training achieved excellent to almost perfect raw (0.88 to 0.94), chance-corrected (kappa 0.72 to 0.88), and chance-independent (Phi 0.74 to 0.89) agreement. We conclude that intensivists without formal consensus training can achieve moderate levels of agreement. Consensus training is necessary to achieve the substantial or almost perfect levels of agreement optimal for the conduct of clinical trials.

The use of the Berlin definition for acute respiratory distress syndrome during infancy and early childhood: multicenter evaluation and expert consensus

“…However, this may be improved through the routine use of a training set of radiographs [26][27][28] and thus this part of our project will help the bedside application of BD.…”

A comparison of the MOS-HIV and SF-12v2 for measuring health-related quality of life of men and women living with HIV/AIDS

“…Weighted kappa (κ) scores as per Fleiss and Cohen [14] were calculated using software by Cyr and Francis [15] in order to determine the chance-corrected agreement between the MOS-HIV and SF-12v2 in assigning individuals to levels of HRQoL. Weighted kappa values were interpreted as follows: less than 0 -- poor agreement; 0 to 0.2 -- slight agreement; 0.2 to 0.4 -- fair agreement; 0.4-0.6 -- moderate agreement; 0.6-0.8 -- substantial agreement; 0.8-1.0 -- almost perfect agreement [16]. …”

Interrater Variability in Identifying Ventilator Associated Pneumonia Using Six Different Definitions

“…Previous studies support the finding that interpretation of chest radiographs can vary between clinicians. 17,18 Treating physicians and radiologists may also interpret chest radiographs differently. 18,19 Furthermore, microbiologic data can be difficult to interpret, as mechanically ventilated patients' airways invariably become colonized with hospital flora.…”

“…17,18 Treating physicians and radiologists may also interpret chest radiographs differently. 18,19 Furthermore, microbiologic data can be difficult to interpret, as mechanically ventilated patients' airways invariably become colonized with hospital flora. ICU patients commonly receive prophylactic, empiric or therapeutic antimicrobials, which can result in change in patients' flora and select out for resistant opportunistic and pathogenic bacteria.…”