Evaluation of the dependence of radiomic features on the machine learning model

Abstract

Background In radiomic studies, several models are often trained with different combinations of feature selection methods and classifiers. The features of the best model are usually considered relevant to the problem, and they represent potential biomarkers. Features selected from statistically similarly performing models are generally not studied. To understand the degree to which the selected features of these statistically similar models differ, 14 publicly available datasets, 8 feature selection methods, and 8 classifiers were used in this retrospective study. For each combination of feature selection and classifier, a model was trained, and its performance was measured with AUC-ROC. The best-performing model was compared to other models using a DeLong test. Models that were statistically similar were compared in terms of their selected features. Results Approximately 57% of all models analyzed were statistically similar to the best-performing model. Feature selection methods were, in general, relatively unstable (0.58; range 0.35–0.84). The features selected by different models varied largely (0.19; range 0.02–0.42), although the selected features themselves were highly correlated (0.71; range 0.4–0.92). Conclusions Feature relevance in radiomics strongly depends on the model used, and statistically similar models will generally identify different features as relevant. Considering features selected by a single model is misleading, and it is often not possible to directly determine whether such features are candidate biomarkers.

A Comprehensive Machine Learning Benchmark Study for Radiomics-Based Survival Analysis of CT Imaging Data in Patients With Hepatic Metastases of CRC

“…Radiomics data are obtained for the whole liver (yellow) and the tumors alone (green). The radiomics features are known to be partially of high correlation 49 . This aspect is displayed for the liver radiomics in Figure 3B via the Pearson correlation matrix.…”

“…The radiomics features are known to be partially of high correlation. 49 This aspect is displayed for the liver radiomics in Figure 3B via the Pearson correlation matrix.…”

Radiomic analysis for early differentiation of lung cancer recurrence from fibrosis in patients treated with lung stereotactic ablative radiotherapy

“…The choice of machine learning methods and feature selection techniques are critical for the overall accuracy of the model. Consequently, we investigated a combination of four widely used machine learning and five feature selection techniques and selected the classification model that achieved the highest AUC score in the test set (Parmar et al 2015, Demircioğlu 2022. In conclusion, we demonstrate the clinical utility of the model with the highest AUC score in differentiating a local recurrence from fibrosis.…”

Predictive value of clot imaging in acute ischemic stroke: A systematic review of artificial intelligence and conventional studies

“…The selection of radiomic features is far from standardization. Research shows that, because feature relevance depends on the used machine learning model [96], the radiomic features cannot become biomarkers. In perspective, multicenter studies comparing the performance of various AI models, using radiomic features and conventional clot related variables, can help render radiomics models effective and accelerate their access into the clinical use.…”