Building Reliable Radiomic Models Using Image Perturbation

Abstract

Radiomic model reliability is a central premise for its clinical translation. Presently, it is assessed using test-retest or external data, which, unfortunately, is often scarce in reality. Therefore, we aimed to develop a novel image perturbation-based method (IPBM) for the first of its kind toward building a reliable radiomic model. We first developed a radiomic prognostic model for head-and-neck cancer patients on a training (70%) and evaluated on a testing (30%) cohort using C-index. Subsequently, we applied the IPBM to CT images of both cohorts (Perturbed-Train and Perturbed-Test cohort) to generate 60 additional samples for both cohorts. Model reliability was assessed using intra-class correlation coefficient (ICC) to quantify consistency of the C-index among the 60 samples in the Perturbed-Train and Perturbed-Test cohorts. Besides, we re-trained the radiomic model using reliable RFs exclusively (ICC>0.75) to validate the IPBM. Results showed moderate model reliability in Perturbed-Train (ICC:0.565, 95%CI:0.518-0.615) and Perturbed-Test (ICC:0.596, 95%CI:0.527-0.670) cohorts. An enhanced reliability of the re-trained model was observed in Perturbed-Train (ICC:0.782, 95%CI:0.759-0.815) and Perturbed-Test (ICC:0.825, 95%CI:0.782-0.867) cohorts, indicating validity of the IPBM. To conclude, we demonstated capability of the IPBM toward building reliable radiomic models, providing community with a novel model reliability assessment strategy prior to prospective evaluation.

Comparing Effectiveness of Image Perturbation and Test-retest Imaging Towards Establishment of Reliable Radiomic Models

“…In addition to the comparisons between perturbation and test-retest, we discovered a positive impact of higher feature repeatability on model reliability, as suggested by the increasing testing AUCs and prediction ICCs under higher ICC thresholds. Our results are consistent with the findings by Teng et al that image perturbation could enhance radiomic model reliability on multiple head-and-neck cancer datasets [14]. A higher model output repeatability is generally guaranteed with increased input repeatabilities when using a linear logistic regression model, as long as the predictability is ensured.…”

“…I(a). We performed 40 image perturbations independently for each patient by random combinations of rotations, translations, and contour randomizations, same as the methdology adopted by Teng et al [14]. Contour randomization was achieved by deforming the original tumor segmentation by a 3-dimensional random displacement field.…”

“…Zwanenburg et al proposed to generate psudo-retest images by random translation, rotation, noise addition and contour randomizations, and demonstrated the similar patterns of feature repeatability to test-retest imaging [13]. Further studies have demonstrated the potential of perturbed images in quantifying radiomic model output reliability and improving the model generalizability and robustness by removing low-repeatable features [14]. Although perturbation methods have been proven to be capable of capturing the majority of non-repeatable features in test-retest images, it is still unknown if image perturbation could replace test-retest imaging in building a reliable radiomic model.…”

“…Model generalizability was evaluated by its discriminatory power on both the training and testing set. Model robustness was quantified by measuring output probability variability on perturbed training, perturbed testing, and test-retest images, similar to the methodology adopted by Teng et al [14]. The comparisons were performed under different feature repeatability thresholds to mimic a wide range of selections of repeatability criteria.…”

Comparing Effectiveness of Image Perturbation and Test-retest Imaging Towards Establishment of Reliable Radiomic Models

“…Zwanenburg et al proposed to generate psudo-retest images by random translation, rotation, noise addition and contour randomizations, and demonstrated the similar patterns of feature repeatability to test-retest imaging [13]. Further studies have demonstrated the potential of perturbed images in quantifying radiomic model output reliability and improving the model generalizability and robustness by removing low-repeatable features [14]. Although perturbation methods have been proven to be capable of capturing the majority of non-repeatable features in test-retest images, it is still unknown if image perturbation could replace test-retest imaging in building a reliable radiomic model.…”