Random projections: Data perturbation for classification problems

Cannings, Timothy I.

doi:10.1002/wics.1499

Cited by 18 publications

(8 citation statements)

References 94 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A key example are random forests (Breiman, 2001), which randomly subsample features. Another key example are random projection ensemble methods, which explicitly aim to reduce dimension by random feature subsampling (Cannings and Samworth, 2017;Gataric et al, 2020), see Cannings (2021) for a review.…”

Section: Random Projection Methodsmentioning

confidence: 99%

Learning Augmentation Distributions using Transformed Risk Minimization

Chatzipantazis¹,

Stefanos²,

Dobriban³

et al. 2021

Preprint

View full text Add to dashboard Cite

Adapting to the structure of data distributions (such as symmetry and transformation invariances) is an important challenge in machine learning. Invariances can be built into the learning process by architecture design, or by augmenting the dataset. Both require a priori knowledge about the exact nature of the symmetries. Absent this knowledge, practitioners resort to expensive and time-consuming tuning. To address this problem, we propose a new approach to learn distributions of augmentation transforms, in a new Transformed Risk Minimization (TRM) framework. In addition to predictive models, we also optimize over transformations chosen from a hypothesis space. As an algorithmic framework, our TRM method is (1) efficient (jointly learns augmentations and models in a single training loop), (2) modular (works with any training algorithm), and (3) general (handles both discrete and continuous augmentations). We theoretically compare TRM with standard risk minimization, and give a PAC-Bayes upper bound on its generalization error. We propose to optimize this bound over a rich augmentation space via a new parametrization over compositions of blocks, leading to the new Stochastic Compositional Augmentation Learning (SCALE) algorithm. We compare SCALE experimentally with prior methods (Fast AutoAugment and Augerino) on CIFAR10/100, SVHN . Additionally, we show that SCALE can correctly learn certain symmetries in the data distribution (recovering rotations on rotated MNIST) and can also improve calibration of the learned model. * Equal Contribution.

show abstract

Section: Random Projection Methodsmentioning

confidence: 99%

Learning Augmentation Distributions using Transformed Risk Minimization

Chatzipantazis¹,

Stefanos²,

Dobriban³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…To apply scagnostics to higher dimensional data sets we derive 10 + √ m random two-dimensional projections of the data set and calculate scagnostics on each (where m is the number of dimensions in the data set). Random projections are used for dimensionality reduction for kmeans clustering [31], and a class of classification methods use multiple random projections to create an ensemble of learners [32,33]. We choose 10 + √ m as having a sufficient number for lower dimensional data sets while growing for higher-dimensional data sets.…”

Section: Extracting Features From Data Setsmentioning

confidence: 99%

New Guidance for Using t-SNE: Alternative Defaults, Hyperparameter Selection Automation, and Comparative Evaluation

Gove¹,

Cadalzo²,

Leiby³

et al. 2022

Preprint

View full text Add to dashboard Cite

We present new guidelines for choosing hyperparameters for t-SNE and an evaluation comparing these guidelines to current ones. These guidelines include a proposed empirically optimum guideline derived from a t-SNE hyperparameter grid search over a large collection of data sets. We also introduce a new method to featurize data sets using graph-based metrics called scagnostics; we use these features to train a neural network that predicts optimal t-SNE hyperparameters for the respective data set. This neural network has the potential to simplify the use of t-SNE by removing guesswork about which hyperparameters will produce the best embedding. We evaluate and compare our neural network-derived and empirically optimum hyperparameters to several other t-SNE hyperparameter guidelines from the literature on 68 data sets. The hyperparameters predicted by our neural network yield embeddings with similar accuracy as the best current t-SNE guidelines. Using our empirically optimum hyperparameters is simpler than following previously published guidelines but yields more accurate embeddings, in some cases by a statistically significant margin. We find that the useful ranges for t-SNE hyperparameters are narrower and include smaller values than previously reported in the literature. Importantly, we also quantify the potential for future improvements in this area: using data from a grid search of t-SNE hyperparameters we find that an optimal selection method could improve embedding accuracy by up to two percentage points over the methods examined in this paper.

show abstract

“…So far we have left unspecified the randomised dimensionality compression scheme to be used. Indeed the general result presented in this section could potentially be applied to any independently randomised ensemble, including random coordinate projections (Ho, 1998;Tian and Feng, 2021), and various sketching methods (Cormode, 2017;Cannings, 2020). For the bound to be useful, we need to be able to control the compressibility function ψ P (k).…”

Section: Main Upper Boundmentioning

confidence: 99%

“…Such technologies open new doors for dealing with massive high dimensional data sets, and inspire new research in areas as diverse as numerical analysis (Halko et al, 2011), statistical methodology (Heinze et al, 2016;Cannings and Samworth, 2017;Tian and Feng, 2021), pattern recognition (Reboredo et al, 2016), clustering (Boutsidis et al, 2015;Biau et al, 2008;Meintrup et al, 2019), optimisation (Pilanci and Wainwright, 2015;Wainwright, 2016, 2017;Derezinski et al, 2020), search based software engineering (Nair et al, 2016), imaging (Lustig et al, 2007;Ye, 2019;Palmer et al, 2015;Bentley et al, 2019), medical research (Peressutti et al, 2015), neuroscience (Arriaga et al, 2015), and computer vision (Jiao et al, 2019). The interested reader may also refer to recent surveys (Gibson et al, 2020), (Cannings, 2020), and references therein.…”

Section: Introductionmentioning

confidence: 99%

Statistical optimality conditions for compressive ensembles

Reeve¹,

Kabán²

2021

Preprint

View full text Add to dashboard Cite

We present a framework for the theoretical analysis of ensembles of low-complexity empirical risk minimisers trained on independent random compressions of high-dimensional data. First we introduce a general distribution-dependent upper-bound on the excess risk, framed in terms of a natural notion of compressibility. This bound is independent of the dimension of the original data representation, and explains the in-built regularisation effect of the compressive approach. We then instantiate this general bound to classification and regression tasks, considering Johnson-Lindenstrauss mappings as the compression scheme. For each of these tasks, our strategy is to develop a tight upper bound on the compressibility function, and by doing so we discover distributional conditions of geometric nature under which the compressive algorithm attains minimax-optimal rates up to at most poly-logarithmic factors. In the case of compressive classification, this is achieved with a mild geometric margin condition along with a flexible moment condition that is significantly more general than the assumption of bounded domain.In the case of regression with strongly convex smooth loss functions we find that compressive regression is capable of exploiting spectral decay with near-optimal guarantees. In addition, a key ingredient for our central upper bound is a high probability uniform upper bound on the integrated deviation of dependent empirical processes, which may be of independent interest.

show abstract

Random projections: Data perturbation for classification problems

Cited by 18 publications

References 94 publications

Learning Augmentation Distributions using Transformed Risk Minimization

Learning Augmentation Distributions using Transformed Risk Minimization

New Guidance for Using t-SNE: Alternative Defaults, Hyperparameter Selection Automation, and Comparative Evaluation

Statistical optimality conditions for compressive ensembles

Contact Info

Product

Resources

About