Random-projection Ensemble Classification

Abstract

We introduce a very general method for high dimensional classification, based on careful combination of the results of applying an arbitrary base classifier to random projections of the feature vectors into a lower dimensional space. In one special case that we study in detail, the random projections are divided into disjoint groups, and within each group we select the projection yielding the smallest estimate of the test error. Our random-projection ensemble classifier then aggregates the results of applying the base classifier on the selected projections, with a data-driven voting threshold to determine the final assignment. Our theoretical results elucidate the effect on performance of increasing the number of projections. Moreover, under a boundary condition that is implied by the sufficient dimension reduction assumption, we show that the test excess risk of the random-projection ensemble classifier can be controlled by terms that do not depend on the original data dimension and a term that becomes negligible as the number of projections increases. The classifier is also compared empirically with several other popular high dimensional classifiers via an extensive simulation study, which reveals its excellent finite sample performance.

Comparison of validation variants by sum of ranking differences and ANOVA

“…Numbers after the abbreviations mean the training set of size of n (a subsample of the data), and then, the remaining data formed the test set. ES k NN, ensemble of subset of k NN classifiers; GP, Gaussian process; k NN, k ‐nearest neighbors; LDA, linear discriminant analysis; NSC, nearest shrunken centroids; OTE, optimal tree ensemble; PenLDA, penalized LDA; QDA, quadratic discriminant analysis; RF, random forest; RP, random projection; SRD, sum of ranking differences; SVM, support vector machine; the number after the abbreviations means “sufficient dimension reduction (SDR5) assumption”…”

“…Cannings and Samworth recently introduced a new technique called “random‐projection ensemble classifier” (RPEC). The authors introduced a general framework for high‐dimensional classification on carefully selected low‐dimensional random projections.…”

“…Here, a multicriteria evaluation is carried out regarding biases and variance. The original authors have used balanced and not‐balanced data sets (eye state detection, free electrons in the ionosphere, Down's syndrome diagnoses, hill‐valley identification, musk identification, and Cardiac arrhythmia diagnoses) . The transpose of tables 3 to 5 of the original publication is suitable for a detailed statistical (SRD) analysis.…”

Estimating the algorithmic variance of randomized ensembles via the bootstrap

“…Similarly, in the case of random forests, each ξ i encodes the points in D * i , as well as randomly chosen sets of features used for training T i . More generally, the representation (1.2) is relevant to other types of randomized ensembles, such as those based on random rotations (Blaser and Fryzlewicz, 2016), random projections (Cannings and Samworth, 2017), or posterior sampling (Ng and Jordan, 2001;Chipman et al, 2010).…”

Sparse Principal Component Analysis via Axis-Aligned Random Projections

“…Algorithms based on random projections have recently been shown to be highly effective for several different problems in high dimensional statistical inference. For instance, in the context of high dimensional classification, Cannings and Samworth () showed that their random projection ensemble classifier that aggregates over projections that yield small estimates of the test error can result in excellent performance. Marzetta et al .…”