Multi-Reader–Multi-Split Annotation of Emphysema in Computed Tomography

Abstract

Emphysema is visible on computed tomography (CT) as low-density lesions representing the destruction of the pulmonary alveoli. To train a machine learning model on the emphysema extent in CT images, labeled image data is needed. The provision of these labels requires trained readers, who are a limited resource. The purpose of the study was to test the reading time, inter-observer reliability and validity of the multi-reader–multi-split method for acquiring CT image labels from radiologists. The approximately 500 slices of each stack of lung CT images were split into 1-cm chunks, with 17 thin axial slices per chunk. The chunks were randomly distributed to 26 readers, radiologists and radiology residents. Each chunk was given a quick score concerning emphysema type and severity in the left and right lung separately. A cohort of 102 subjects, with varying degrees of visible emphysema in the lung CT images, was selected from the SCAPIS pilot, performed in 2012 in Gothenburg, Sweden. In total, the readers created 9050 labels for 2881 chunks. Image labels were compared with regional annotations already provided at the SCAPIS pilot inclusion. The median reading time per chunk was 15 s. The inter-observer Krippendorff’s alpha was 0.40 and 0.53 for emphysema type and score, respectively, and higher in the apical part than in the basal part of the lungs. The multi-split emphysema scores were generally consistent with regional annotations. In conclusion, the multi-reader–multi-split method provided reasonably valid image labels, with an estimation of the inter-observer reliability.

Machine learning slice-wise whole-lung CT emphysema score correlates with airway obstruction

“…Concerning machine learning CT emphysema assessment, the validation cohort has not been previously reported. Related to the present work, the subjects in the development cohort have previously been reported by Vikgren et al [ 17 ] and Lidén et al [ 18 ].…”

“…Detailed slice-wise emphysema annotations were acquired in a three-step process including multiple readers in the first (26 readers) and third steps (16 readers) (see supplemental Fig S1 – S3 ). In the first step, each centimeter of each lung was classified according to a 4-degree emphysema scale as previously described [ 18 ]. The second step consisted of median z -direction filtering of the 4-degree annotations and the third step was a refinement algorithm, increasing the granularity of the emphysema labels from a 4-degree scale to 10 degrees (see Appendix A for details).…”