False Discovery Rates in PET and CT Studies with Texture Features: A Systematic Review

Abstract

PurposeA number of recent publications have proposed that a family of image-derived indices, called texture features, can predict clinical outcome in patients with cancer. However, the investigation of multiple indices on a single data set can lead to significant inflation of type-I errors. We report a systematic review of the type-I error inflation in such studies and review the evidence regarding associations between patient outcome and texture features derived from positron emission tomography (PET) or computed tomography (CT) images.MethodsFor study identification PubMed and Scopus were searched (1/2000–9/2013) using combinations of the keywords texture, prognostic, predictive and cancer. Studies were divided into three categories according to the sources of the type-I error inflation and the use or not of an independent validation dataset. For each study, the true type-I error probability and the adjusted level of significance were estimated using the optimum cut-off approach correction, and the Benjamini-Hochberg method. To demonstrate explicitly the variable selection bias in these studies, we re-analyzed data from one of the published studies, but using 100 random variables substituted for the original image-derived indices. The significance of the random variables as potential predictors of outcome was examined using the analysis methods used in the identified studies.ResultsFifteen studies were identified. After applying appropriate statistical corrections, an average type-I error probability of 76% (range: 34–99%) was estimated with the majority of published results not reaching statistical significance. Only 3/15 studies used a validation dataset. For the 100 random variables examined, 10% proved to be significant predictors of survival when subjected to ROC and multiple hypothesis testing analysis.ConclusionsWe found insufficient evidence to support a relationship between PET or CT texture features and patient survival. Further fit for purpose validation of these image-derived biomarkers should be supported by appropriate biological and statistical evidence before their association with patient outcome is investigated in prospective studies.

Hybrid Imaging: Instrumentation and Data Processing

“…Textural parameters are strongly influenced by the choice of image reconstruction, voxel size and post processing steps (e.g., Gaussian filtering). Furthermore, the choice and definitions of the textural analysis metrics vary widely and the evaluation techniques differ substantially, thus, rendering a comparison between studies almost impossible [260,297]. The need for standardization efforts in the field of textural analyses has been highlighted previously [260,298,299], and efforts are currently under way to address this challenge [260,[300][301][302].…”

Texture analysis of medical images for radiotherapy applications

“…Moreover, in this phase, the sample size should be defined by considering the high number of image-based features that can be estimated from the acquired images. In general, this issue has not been correctly faced, since the number of features is often greater than the number of considered patients, and consequently, a high risk of false-positive discovery rate can be easily found, as recently highlighted by Yip and Aerts 25 and deeply discussed in Chalkidou et al 105 Regarding the statistical data analysis, it is evident that texture analysis can provide a very complex and large set of data, which can present high correlations among them. It is thus necessary to reduce the number of features to avoid the risk of overfitting analysis and to build a classifier or prediction model.…”

“…Another factor that can impact on the classification performance is the choice of the optimal cut-off to stratify patients into a binary model. 105 This choice is highly dependent on the studied data set, and thus it is difficult to be applied in external populations. 25 Another issue that should be faced during the study design phase is how to generalize and validate the radiomic signature found in the assessed patients population.…”

Understanding Changes in Tumor Texture Indices in PET: A Comparison Between Visual Assessment and Index Values in Simulated and Patient Data

“…These indices, if proven, could be used to reliably identify the existence of subpopulations of cells with distinct genomic alterations and could guide the choice of treatment, especially for targeted therapeutics (2). In PET, several retrospective studies suggest that texture indices reflect tumor heterogeneity and predict treatment response or patient survival whereas other studies underline the limitations of these indices (3)(4)(5)(6)(7)(8)(9). In addition, the interpretation of texture index values derived from PET images has never been reported, and texture indices have been investigated only in retrospective studies (10).…”