Improved Guarantees and a Multiple-descent Curve for Column Subset Selection and the Nystrom Method (Extended Abstract)

Abstract

The Column Subset Selection Problem (CSSP) and the Nystrom method are among the leading tools for constructing interpretable low-rank approximations of large datasets by selecting a small but representative set of features or instances. A fundamental question in this area is: what is the cost of this interpretability, i.e., how well can a data subset of size k compete with the best rank k approximation? We develop techniques which exploit spectral properties of the data matrix to obtain improved approximation guarantees which go beyond the standard worst-case analysis. Our approach leads to significantly better bounds for datasets with known rates of singular value decay, e.g., polynomial or exponential decay. Our analysis also reveals an intriguing phenomenon: the cost of interpretability as a function of k may exhibit multiple peaks and valleys, which we call a multiple-descent curve. A lower bound we establish shows that this behavior is not an artifact of our analysis, but rather it is an inherent property of the CSSP and Nystrom tasks. Finally, using the example of a radial basis function (RBF) kernel, we show that both our improved bounds and the multiple-descent curve can be observed on real datasets simply by varying the RBF parameter.

Connections and Equivalences between the Nyström Method and Sparse Variational Gaussian Processes

“…Similarly, the recent advances in using and understanding the Nyström method (Williams and Seeger, 2001), which is one of the most popular sparse approximations in kernel methods, have been made independently to those of sparse GP approximations. The majority of these advances focus on an efficient approximation of the kernel matrix (e.g., Drineas and Mahoney, 2005;Belabbas and Wolfe, 2009;Gittens and Mahoney, 2016;Derezinski et al, 2020) or empirical risk minimization in the RKHS with a reduced basis (e.g, Bach, 2013;El Alaoui and Mahoney, 2015;Rudi et al, 2015Rudi et al, , 2017Meanti et al, 2020). This separation of two lines of research are arguably due to the difference in the notations and modeling philosophies of GPs and kernel methods.…”

Local optimisation of Nyström samples through stochastic gradient descent

“…The initial Nyström samples  (0) we considered were draw uniformly at random without replacement; while our experiments suggest that the local minima of the radial SKD often induce approximations of comparable quality, the use of more efficient initialisation strategies may be investigated (see e.g. [3,4,11,13,18]).…”

“…For a Nyström sample  ∈ of size ∈ ℕ, the matrix ̂ () is of rank at most . Following [4,10], to further assess the efficiency of the approximation of induced by , we introduce the approximation factors…”

“…In Data Science, the Nyström method refers to a specific technique for the low-rank approximation of symmetric positive-semidefinite (SPSD) matrices; see e.g. [4,5,10,11,18]. Given an × SPSD matrix , with ∈ ℕ, the Nyström method consists of selecting a sample of ∈ ℕ columns of , generally with ≪ , and next defining a low-rank approximation ̂ of based on this sample of columns.…”

Optimal Subsampling for Large Sample Ridge Regression

“…Kumar et al (2012) explored the sampling approach for the column subset selection problem by the Nyström method. Derezinski et al (2020) recently provided an improved theoretical guarantee for low-rank approximations of large datasets. Another popular idea in machine learning is coreset, which constructs estimators based on sub-data.…”