Spectral methods in machine learning and new strategies for very large datasets

Abstract

Spectral methods are of fundamental importance in statistics and machine learning, because they underlie algorithms from classical principal components analysis to more recent approaches that exploit manifold structure. In most cases, the core technical problem can be reduced to computing a low-rank approximation to a positive-definite kernel. For the growing number of applications dealing with very large or high-dimensional datasets, however, the optimal approximation afforded by an exact spectral decomposition is too costly, because its complexity scales as the cube of either the number of training examples or their dimensionality. Motivated by such applications, we present here 2 new algorithms for the approximation of positive-semidefinite kernels, together with error bounds that improve on results in the literature. We approach this problem by seeking to determine, in an efficient manner, the most informative subset of our data relative to the kernel approximation task at hand. This leads to two new strategies based on the Nyström method that are directly applicable to massive datasets. The first of these-based on sampling-leads to a randomized algorithm whereupon the kernel induces a probability distribution on its set of partitions, whereas the latter approach-based on sorting-provides for the selection of a partition in a deterministic way. We detail their numerical implementation and provide simulation results for a variety of representative problems in statistical data analysis, each of which demonstrates the improved performance of our approach relative to existing methods.statistical data analysis | kernel methods | low-rank approximation S pectral methods hold a central place in statistical data analysis. Indeed, the spectral decomposition of a positive-definite kernel underlies a variety of classical approaches such as principal components analysis (PCA), in which a low-dimensional subspace that explains most of the variance in the data is sought; Fisher discriminant analysis, which aims to determine a separating hyperplane for data classification; and multidimensional scaling (MDS), used to realize metric embeddings of the data. Moreover, the importance of spectral methods in modern statistical learning has been reinforced by the recent development of several algorithms designed to treat nonlinear structure in data-a case where classical methods fail. Popular examples include isomap (1), spectral clustering (2), Laplacian (3) and Hessian (4) eigenmaps, and diffusion maps (5). Though these algorithms have different origins, each requires the computation of the principal eigenvectors and eigenvalues of a positive-definite kernel.Although the computational cost (in both space and time) of spectral methods is but an inconvenience for moderately sized datasets, it becomes a genuine barrier as data sizes increase and new application areas appear. A variety of techniques, spanning fields from classical linear algebra to theoretical computer science (6), have been proposed to trade off analysis precision ...

Diffusion Maps for dimensionality reduction and visualization of meteorological data

“…Observe that in principle, sample size coincides with the dimension of the similarity and, hence, transition matrices, which may make its eigenanalysis computationally unfeasible for very large data sets. We will not deal with the issue of the complexity of DM's eigenanalysis in this work but point out that the usual approach is to apply an adequate subsampling of the original data [14]. In turn, this requires a mechanism to extend the embedding computed on that subsample to the other data points not considered or, more generally, new, unseen patterns.…”

Iterative numerical methods for sampling from high dimensional Gaussian distributions

“…For very high dimension the storage capacity seems very large for this method, but this can be overcome using a so-called two-pass strategy described in Frommer and Simoncini (2008). In the machine learning literature, Belabbas and Wolfe (2009) used iterative methods to approximate the eigenvalues, and capture the most important feature of the Gaussian process. Their approach seems to work well for moderate dimensions, but it is unclear what its properties are, and how to tune this method in high dimensions.…”

A New Anticorrelation-Based Spectral Clustering Formulation

“…This is a well-known fact which has been continually emphasized over the past decade [19,6]. Approaches dealing with this paradigm range from exploiting the sparsity, subsampling of an image or similarity matrix to the low-rank approximation methods such as Nyström algorithm [21,23]. We infer that this is still an open problem from the most recent work by Chen et al in [25] where the team of researchers presents a parallel HPC implementation of SC.…”