Test–Retest Data for Radiomics Feature Stability Analysis: Generalizable or Study-Specific?

Abstract

Radiomics is an objective method for extracting quantitative information from medical images. However, in radiomics, standardization, overfitting, and generalization are major challenges to be overcome. Test–retest experiments can be used to select robust radiomic features that have minimal variation. Currently, it is unknown whether they should be identified for each disease (disease specific) or are only imaging device-specific (computed tomography [CT]-specific). Here, we performed a test–retest analysis on CT scans of 40 patients with rectal cancer in a clinical setting. Correlation between radiomic features was assessed using the concordance correlation coefficient (CCC). In total, only 9/542 features have a CCC > 0.85. Furthermore, results were compared with the test–retest results on CT scans of 27 patients with lung cancer with a 15-minute interval. Results show that 446/542 features have a higher CCC for the test–retest analysis of the data set of patients with lung cancer than for patients with rectal cancer. The importance of controlling factors such as scanners, imaging protocol, reconstruction methods, and time points in a radiomics analysis is shown. Moreover, the results imply that test–retest analyses should be performed before each radiomics study. More research is required to independently evaluate the effect of each factor.

Radiomics in medical imaging—“how-to” guide and critical reflection

“…Although these studies are relevant to create awareness of the influencing factors, it should be noted that the information is often not directly helpful to future studies. The reproducibility of radiomic features is not necessarily generalizable to different disease sites, modalities, or scanners, e.g., robust features in one disease site are not necessarily robust in another disease site [32]. Moreover, in case robust radiomic features are assessed using cut-off values of correlation coefficients, one should be aware that these cut-offs are often arbitrarily chosen and the number of "robust" features depend on the number of subjects involved.…”

Radiomics as a personalized medicine tool in lung cancer: Separating the hope from the hype

“…Image acquisition A Different scanners and acquisition protocols affect feature reproducibility [79][80][81][82][83][84][85][86][87][88][89][90][91] Image phantoms on different scanners to provide baseline [79], establish credibility of scanners and protocols [84], catalogue reproducible features [86,90], model a correction algorithm [89] [98,[110][111][112] Normalization of features to volume [98], bit depth resampling [110], feature redesign [110], more robust statistics to check added value of radiomics signatures [111]. Test re-test H Radiomic features may not be repeatable over multiple measurements [113][114][115], repeatable features are not generalizable to other disease sites [116].…”

“…Test-retest data acquisition [113,116], use of multiple 4D phases [113,115], use of simulated retest by image perturbation [114].…”

“…As highlighted by several studies, [113,116] and by consensus statements on imaging biomarkers [12], radiomics studies usually lack an assessment of the signatures' single centre repeatability or multicentre reproducibility. The use of test-retest datasets in which multiple images of the same subjects or phantom have been acquired in quick succession have been proposed as a means to assess repeatability [113,116]. Alternative options include the use of multiple 4D image phases [113] and the simulation of retest data by image perturbation [114] where test-retest data are not available.…”

Radiomics and liquid biopsy in oncology: the holons of systems medicine

“…Overfitting occurs when a model begins to memorise training data rather than learning to generalise from a trend. The main cause of overfitting in radiomics is the application of too many features, which become redundant and irrelevant (so-called noise); the excessive number of features can be reduced by test–retest studies that enable the selection of only those robust features that provide repeatable and reproducible measurements [ 9 , 26 , 27 ].…”