Improving SNR and Reducing Training Time of Classifiers in Large Datasets via Kernel Averaging

Abstract

Kernel methods are of growing importance in neuroscience research. As an elegant extension of linear methods, they are able to model complex non-linear relationships. However, since the kernel matrix grows with data size, the training of classifiers is computationally demanding in large datasets. Here, a technique developed for linear classifiers is extended to kernel methods: In linearly separable data, replacing sets of instances by their averages improves signal-to-noise ratio (SNR) and reduces data size. In kernel methods, data is linearly non-separable in input space, but linearly separable in the high-dimensional feature space that kernel methods implicitly operate in. It is shown that a classifier can be efficiently trained on instances averaged in feature space by averaging entries in the kernel matrix. Using artificial and publicly available data, it is shown that kernel averaging improves classification performance substantially and reduces training time, even in non-linearly separable data.

MVPA-Light: A Classification and Regression Toolbox for Multi-Dimensional Data

“…For linear classifiers, this approach has been explored by (Cichy et al, 2015;Cichy and Pantazis, 2017). Recently, it has been generalized to non-linear kernel methods (Treder, 2018). Either approach can be used in the toolbox by adding the operation average_samples or average_kernel to the preprocessing pipeline.…”

“…That is, they are performed on the training data first and subsequently applied to the test data using parameters estimated from the training data (Lemm et al, 2011;Varoquaux et al, 2017). Currently implemented functions include PCA, sample averaging (Cichy and Pantazis, 2017), kernel averaging (Treder, 2018), and under-/oversampling for unbalanced data. Preprocessing pipelines are defined by adding the cfg.preprocess parameter.…”

Impact of Satellite-Derived Land Cover Resolution Using Machine Learning and Hydrological Simulations

“…The most often used classifier, random forest (RF), builds an ensemble classifier by combining many CART trees [33]. RF builds several decision trees using a random selection of training datasets and parameters.…”