Assessing the performance of statistical validation tools for megavariate metabolomics data

Abstract

Statistical model validation tools such as cross-validation, jack-knifing model parameters and permutation tests are meant to obtain an objective assessment of the performance and stability of a statistical model. However, little is known about the performance of these tools for megavariate data sets, having, for instance, a number of variables larger than 10 times the number of subjects. The performance is assessed for megavariate metabolomics data, but the conclusions also carry over to proteomics, transcriptomics and many other research areas. Partial least squares discriminant analyses models were built for several LC-MS lipidomic training data sets of various numbers of lean and obese subjects. The training data sets were compared on their modelling performance and their predictability using a 10-fold cross-validation, a permutation test, and test data sets. A wide range of cross-validation error rates was found (from 7.5% to 16.3% for the largest trainings set and from 0% to 60% for the smallest training set) and the error rate increased when the number of subjects decreased. The test error rates varied from 5% to 50%. The smaller the number of subjects compared to the number of variables, the less the outcome of validation tools such as cross-validation, jack-knifing model parameters and permutation tests can be trusted. The result depends crucially on the specific sample of subjects that is used for modelling. The validation tools cannot be used as warning mechanism for problems due to sample size or to representativity of the sampling.

Effects of hypoxia in the gills of the Manila clam Ruditapes philippinarum using NMR-based metabolomics

“…The PLS-DA included a measured matrix X and a response matrix Y consisting of dummy variables. The quality of the PLS-DA model was assessed using a leave-one-out cross-validation method (Rubingh et al, 2006). The latent variables producing increases in Q 2 were considered as significant components, where:…”

Metabolic responses in gills of Manila clam Ruditapes philippinarum exposed to copper using NMR-based metabolomics

“…In PLS-DA, the X matrix is the measured matrix, i.e., the NMR data, and the Y matrix is composed of dummy variables (represented by ones and zeros) that indicate the class for each treatment . The prediction accuracy of the PLS-DA model was assessed using cross-validation with leave-one-out (Rubingh et al, 2006;Westerhuis et al, 2008). A Q 2 score > 0.08 indicates that the model classification is significantly better than chance, while a score greater than 0.4 indicates that the model is practically robust (Lindon et al, 1999).…”

Spectral pre and post processing for infrared and Raman spectroscopy of biological tissues and cells

“…Other model validation techniques commonly employed include Leave one out cross validation or up to 10-fold cross validation [60,61]. Pérez-Guaita et al [62] more recently evaluated the use of permutation testing, commonly used in metabolomics [63] and proteomics [64], which employs a random reallocation of class labels in order to establish the statistical significance of a cross-validation figure of merit of a classifier. Ultimately, however, the validation of the integrated techniques of spectroscopy and multivariate classifiers will have to comply with the rigours of the clinical environment, including large scale blind datasets and randomised trials [1].…”