Merits of random forests emerge in evaluation of chemometric classifiers by external validation

Abstract

Real-world applications will inevitably entail divergence between samples on which chemometric classifiers are trained and the unknowns requiring classification. This has long been recognized, but there is a shortage of empirical studies on which classifiers perform best in 'external validation' (EV), where the unknown samples are subject to sources of variation relative to the population used to train the classifier. Survey of 286 classification studies in analytical chemistry found only 6.6% that stated elements of variance between training and test samples. Instead, most tested classifiers using hold-outs or resampling (usually cross-validation) from the same population used in training. The present study evaluated a wide range of classifiers on NMR and mass spectra of plant and food materials, from four projects with different data properties (e.g., different numbers and prevalence of classes) and classification objectives. Use of cross-validation was found to be optimistic relative to EV on samples of different provenance to the training set (e.g., different genotypes, different growth conditions, different seasons of crop harvest). For classifier evaluations across the diverse tasks, we used ranks-based non-parametric comparisons, and permutation-based significance tests. Although latent variable methods (e.g., PLSDA) were used in 64% of the surveyed papers, they were among the less successful classifiers in EV, and orthogonal signal correction was counterproductive. Instead, the best EV performances were obtained with machine learning schemes that coped with the high dimensionality (914-1898 features). Random forests confirmed their resilience to high dimensionality, as best overall performers on the full data, despite being used in only 4.5% of the surveyed papers. Most other machine learning classifiers were improved by a feature selection filter (ReliefF), but still did not out-perform random forests.

Feature Selection Methods for Early Predictive Biomarker Discovery Using Untargeted Metabolomic Data

“…Indeed, these top features were ranked among the 10 first ones with each techniques. This was also observed in previous published studies (Menze et al, 2009; Guo and Balasubramanian, 2012; Chen et al, 2013; Scott et al, 2013) with high dimension datasets from omics data. In particular, RF was shown to be robust to noise and outliers, and a technique of choice to avoid overfitting.…”

“…Its main advantage, as reported in the literature (Ho, 1998; Liaw and Wiener, 2002; Biau, 2012; Hapfelmeier et al, 2014), includes principally its power to deal with over-fitting and missing data, as well as its capacity to handle large datasets without variable elimination in terms of feature selection (Menze et al, 2009). It was successfully applied as a biomarker selection tool for metabolomic data analysis in several studies (Chen et al, 2013; Scott et al, 2013; Gromski et al, 2015), especially due to its resilience to high dimensionality data, insensitivity to noise, and resistance to overfitting, etc. Nevertheless, it generates different results, contrary to SVM which delivers a unique solution.…”

“…Therefore, these alternative approaches may be useful tools for generating various models through data reduction and feature selection, as well as providing more accurate results. However, different studies showed that the choice of the appropriate algorithms and workflow is highly dependent of the dataset characteristics and the objective of the data mining process (Scott et al, 2013). …”

Exploring bitterness of traditional Chinese medicine samples by potentiometric electronic tongue and by capillary electrophoresis and liquid chromatography coupled to UV detection

“…In order to confirm the validity of PLS regression models we employed cross-validation, independent test sets [32,34] and permutation testing [33]. In case of HPLC and CE data processing variable selection [35] was performed on the basis of regression coefficients of the variables in the models.…”

Large-scale identification of potential drug targets based on the topological features of human protein–protein interaction network

“…Compared with other machine learning algorithm, RF has many specific properties, such as computational efficiency on large dataset, outstanding performance in the prediction accuracy and good estimation of important variables. During the past decades, RF has been widely used in the field of analytical chemistry and chemometrics [31][32][33][34]. For detailed description of RF, see Ref.…”