Vulnerabilities of radiomic signature development: The need for safeguards

Abstract

Refinement of radiomic results and methodologies is required to ensure progression of the field. In this work, we establish a set of safeguards designed to improve and support current radiomic methodologies through detailed analysis of a radiomic signature. Methods: A radiomic model (MW2018) was fitted and externally validated using features extracted from previously reported lung and head and neck (H&N) cancer datasets using gross-tumour-volume contours, as well as from images with randomly permuted voxel index values; i.e. images without meaningful texture. To determine MW2018's added benefit, the prognostic accuracy of tumour volume alone was calculated as a baseline. Results: MW2018 had an external validation concordance index (c-index) of 0.64. However, a similar performance was achieved using features extracted from images with randomized signal intensities (c-index = 0.64 and 0.60 for H&N and lung, respectively). Tumour volume had a c-index = 0.64 and correlated strongly with three of the four model features. It was determined that the signature was a surrogate for tumour volume and that intensity and texture values were not pertinent for prognostication. Conclusion: Our experiments reveal vulnerabilities in radiomic signature development processes and suggest safeguards that can be used to refine methodologies, and ensure productive radiomic development using objective and independent features.

T2w‐MRI signal normalization affects radiomics features reproducibility

“…This is in line with recent publications that highlighted some vulnerabilities in the radiomic signature development, related to the risk of including features that are mainly correlated to the volume in prediction models. 35,38 Reproducibility of radiomic features was here assessed using the ICC metric, able to combine information about the degree of correlation and agreement between measurements. 34 This coefficient is one of the most adopted for the estimation of repeatability and reproducibility of radiomic indices, as reported in Traverso et al 8 In fact, it has been used also in the latest work considering T2w-MRI.…”

“…The Spearman correlation between radiomic features and the ROI volume was assessed, since it has been reported that many features intrinsically embed volume information. 35 In this way, it is possible to discard the highly correlated ones (significant P-value < 0.05, after Bonferroni correction for multiple comparisons), in order to consider only features that embed information related to the texture. In addition, the assessment of inter-correlations between features was also performed using Spearman correlation (significant P-value < 0.05 after Bonferroni correction for multiple comparisons).…”

Image-Based Molecular Phenotyping of Pancreatic Ductal Adenocarcinoma

“…Of these, features with missing values, all-null values, zero variance, features unstable to between-observer segmentation or to segmentation and re-segmentation (based on an intra-class correlation coefficient [13] below 0.9) were eliminated from the analysis. The remaining 161 features were normalized by tumor volume (calculated by PyRadiomics as mesh volume) as suggested by [14]. A Random Forest machine-learning algorithm [15] was used to model the features using the settings detailed in the Supplementary Materials with target labels of QM (KRT81+/HNF1a-) or non-QM (KRT81-/HNF1a-or KRT81-/HNF1a+).…”

Sensitivity of radiomic features to inter-observer variability and image pre-processing in Apparent Diffusion Coefficient (ADC) maps of cervix cancer patients

“…The bin width was chosen to produce between 30 and 128 bin counts as recommended by Tixier et al [18]. The normalization S100 (Table 1) with bin width 5 (referred as S100BW5) produced a median of 25 bins (range 10-40) and thus was excluded from subsequent analysis as per [14].…”

“…To extensively assess feature reproducibility, there is a need to investigate the sensitivity of radiomic features to inter-observer variability and image pre-processing in ADC. Finally, correlations between radiomic features (many of which seem to be a surrogate of tumor volume [14]) and tumor volume should also be investigated, to avoid the risk of introducing redundant information and over-fitting in radiomics-based prediction models.…”