Assessing the inter-observer variability of Computer-Aided Nodule Assessment and Risk Yield (CANARY) to characterize lung adenocarcinomas

Abstract

Lung adenocarcinoma (ADC), the most common lung cancer type, is recognized increasingly as a disease spectrum. To guide individualized patient care, a non-invasive means of distinguishing indolent from aggressive ADC subtypes is needed urgently. Computer-Aided Nodule Assessment and Risk Yield (CANARY) is a novel computed tomography (CT) tool that characterizes early ADCs by detecting nine distinct CT voxel classes, representing a spectrum of lepidic to invasive growth, within an ADC. CANARY characterization has been shown to correlate with ADC histology and patient outcomes. This study evaluated the inter-observer variability of CANARY analysis. Three novice observers segmented and analyzed independently 95 biopsy-confirmed lung ADCs from Vanderbilt University Medical Center/Nashville Veterans Administration Tennessee Valley Healthcare system (VUMC/TVHS) and the Mayo Clinic (Mayo). Inter-observer variability was measured using intra-class correlation coefficient (ICC). The average ICC for all CANARY classes was 0.828 (95% CI 0.76, 0.895) for the VUMC/TVHS cohort, and 0.852 (95% CI 0.804, 0.901) for the Mayo cohort. The most invasive voxel classes had the highest ICC values. To determine whether nodule size influenced inter-observer variability, an additional cohort of 49 sub-centimeter nodules from Mayo were also segmented by three observers, with similar ICC results. Our study demonstrates that CANARY ADC classification between novice CANARY users has an acceptably low degree of variability, and supports the further development of CANARY for clinical application.

Computed Tomography–Based Score Indicative of Lung Cancer Aggression (SILA) Predicts the Degree of Histologic Tissue Invasion and Patient Survival in Lung Adenocarcinoma Spectrum

“…Previously, CANARY has been validated to have very low degree of interobserver variability, even among novice users. 18 However, the interobserver reproducibility of SILA computation with use of the aforementioned semiautomated seed-growing algorithm was evaluated by Bland-Altman plot of two observers independently segmenting 45 separate AC nodules. The interrater reliability of the categorization based on SILA thresholds was also assessed by using the linear weighted k.…”

Computer Aided Nodule Analysis and Risk Yield (CANARY) characterization of adenocarcinoma: radiologic biopsy, risk stratification and future directions

“…A Kappa score of 0.61-0.80 signifies significant agreement and 0.81-1.00 signifies perfect agreement. Our group concluded that with adequate training, CANARY can be widely applied with reproducible results (23). Additional internal validation among three independent users (B Bartholmai, R Karwoski, S Rajagopalan) for 283 adenocarcinomas from Mayo historical cases again showed excellent ICC for all exemplars and the summed V-I-R-O category ( Table 2).…”

Evaluation of Computer-Aided Nodule Assessment and Risk Yield (CANARY) in Korean patients for prediction of invasiveness of ground-glass opacity nodule

“…In this study, we evaluated the versatility of CANARY by applying Korean patients. CANARY has been developed through constant validation on various users and institutions, but still has a limitation that it has only been verified in American and European patients [4][5][6][7][8][9]. This study is of great significance in that it was able to validation CANARY on Asian patients for the first time.…”

“…Computer-Aided Nodule Assessment and Risk Yield (CANARY), which is based on a machine learning technique, is one of the most advanced types of software. It has been steadily improved, and its performance has been verified through previous studies [4][5][6][7][8][9]. CANARY divides semi-auto segmented GGN regions of interest into nine distinct exemplars based on radiomic features and clusters them into three separate groups for risk stratification.…”