Learning Adaptive Random Features

Abstract

Distributed learning and random projections are the most common techniques in large scale nonparametric statistical learning. In this paper, we study the generalization properties of kernel ridge regression using both distributed methods and random features. Theoretical analysis shows the combination remarkably reduces computational cost while preserving the optimal generalization accuracy under standard assumptions. In a benign case, O( √ N ) partitions and O( √ N ) random features are sufficient to achieve O(1/N ) learning rate, where N is the labeled sample size. Further, we derive more refined results by using additional unlabeled data to enlarge the number of partitions and by generating features in a data-dependent way to reduce the number of random features.Preprint. Under review.

Divide-and-Conquer Learning with Nyström: Optimal Rate and Algorithm

“…To overcome the computational and memory bottleneck of KRLS, practical algorithms are developed, including Nyström approach (Rudi, Carratino, and Rosasco 2017;Camoriano et al 2016) and divide-and-conquer (Zhang, Duchi, and Wainwright 2013;Li, Liu, and Wang 2019) of which statistical properties are well studied. Nyström (Rudi, Camoriano, and Rosasco 2015;Camoriano et al 2016) tactfully constructs some small-scale matrices, by sampling the dataset, to approximate the raw kernel matrix so that the time and space complexity can make a sudden drop.…”

“…In this section, for exploring the generalization ability, we firstly introduce four standard assumptions, which are widely used in statistical learning of squared loss (Smale and Zhou 2007;Caponnetto and Vito 2007;Rudi, Carratino, and Rosasco 2017;Li, Liu, and Wang 2019). Under the basic assumptions, the theoretical bound of the proposed algorithm is provided, which is the same as that of the exact Kernel Regularized Least Squares (KRLS).…”

“…. Although Random Features (Rudi, Camoriano, and Rosasco 2016) and DC-Random Features (Li, Liu, and Wang 2019) can also obtain the same optimal convergence rate as DC-NY, their corresponding…”

“…It computes KRLS in parallel by dividing into some subsets and then merge the result from each subset to get the final approximation. Recently, combinations of those accelerated algorithms have also captured a lot of attention, of which learning properties have been explored including the combination of divide-and-conquer and SGD (Lin and Cevher 2018) and the combination of divide-andconquer and random features (Li, Liu, and Wang 2019). Even though the state-of-the-art KRLS estimates can preserve the same optimal statistical accuracy of exact KRLS, the computational requirements of them are still prohibitive faced with large-scale datasets, namely, there are no corresponding computational lower bounds.…”

“…The code is from website 3 . (3) DC-RF: it represents the algorithm(Li, Liu, and Wang 2019): combining Random Features with Divide-and-Conquer. The code is from website 4 .…”