Data-driven Random Fourier Features using Stein Effect

Chang, Wei‐Cheng; Li, Chun-Liang; Yang, Yiming; Póczos, Barnabás

doi:10.24963/ijcai.2017/207

Cited by 15 publications

(6 citation statements)

References 1 publication

Supporting

Mentioning

Contrasting

Order By: Relevance

“…weighted random features: [33], [82] for RFF, [26] for QMC, [27] for GQ kernel alignment: KA-RFF [83] and KP-RFF [45] compressed low-rank approximation: CLR-RFF [47] kernel learning by random features…”

Section: Taxonomy Of Random Features Based Algorithmsmentioning

confidence: 99%

“…ii) Re-weighted random feature selection: Here the basic idea is to re-weight the random features by solving a constrained optimization problem. Examples of this approach include weighted RFF [33], [82], weighted QMC [26], and weighted GQ [27]. Note that these algorithms directly learn the weights of pre-given random features.…”

Section: Taxonomy Of Random Features Based Algorithmsmentioning

confidence: 99%

See 1 more Smart Citation

Random Features for Kernel Approximation: A Survey on Algorithms, Theory, and Beyond

Liu

Huang

Chen

et al. 2022

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

Random features is one of the most popular techniques to speed up kernel methods in large-scale problems. Related works have been recognized by the NeurIPS Test-of-Time award in 2017 and the ICML Best Paper Finalist in 2019. The body of work on random features has grown rapidly, and hence it is desirable to have a comprehensive overview on this topic explaining the connections among various algorithms and theoretical results. In this survey, we systematically review the work on random features from the past ten years. First, the motivations, characteristics and contributions of representative random features based algorithms are summarized according to their sampling schemes, learning procedures, variance reduction properties and how they exploit training data. Second, we review theoretical results that center around the following key question: how many random features are needed to ensure a high approximation quality or no loss in the empirical/expected risks of the learned estimator. Third, we provide a comprehensive evaluation of popular random features based algorithms on several large-scale benchmark datasets and discuss their approximation quality and prediction performance for classification. Last, we discuss the relationship between random features and modern over-parameterized deep neural networks (DNNs), including the use of high dimensional random features in the analysis of DNNs as well as the gaps between current theoretical and empirical results. This survey may serve as a gentle introduction to this topic, and as a users' guide for practitioners interested in applying the representative algorithms and understanding theoretical results under various technical assumptions. We hope that this survey will facilitate discussion on the open problems in this topic, and more importantly, shed light on future research directions.

show abstract

Section: Taxonomy Of Random Features Based Algorithmsmentioning

confidence: 99%

Section: Taxonomy Of Random Features Based Algorithmsmentioning

confidence: 99%

Random Features for Kernel Approximation: A Survey on Algorithms, Theory, and Beyond

Liu

Huang

Chen

et al. 2022

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

show abstract

“…Similarly, sampling from an importance-weighted distribution may also be used in low-rank matrix approximation, i.e., column sampling, but algorithms in the setting of the column sampling [45][46][47] are not applicable to RFs [9]. Quasi-Monte Carlo techniques [48,49] can also improve M , but it is unknown whether they can achieve minimal M . In contrast, our algorithm achieves minimal M within feasible runtime.…”

Section: Generalization Property and Runtime Of Classification With O...mentioning

confidence: 99%

Exponential Error Convergence in Data Classification with Optimized Random Features: Acceleration by Quantum Machine Learning

Yamasaki¹,

Sonoda²

2021

Preprint

View full text Add to dashboard Cite

Random features are a central technique for scalable learning algorithms based on kernel methods. A recent work has shown that an algorithm for machine learning by quantum computer, quantum machine learning (QML), can exponentially speed up sampling of optimized random features, even without imposing restrictive assumptions on sparsity and low-rankness of matrices that had limited applicability of conventional QML algorithms; this QML algorithm makes it possible to significantly reduce and provably minimize the required number of features for regression tasks. However, a major interest in the field of QML is how widely the advantages of quantum computation can be exploited, not only in the regression tasks. We here construct a QML algorithm for a classification task accelerated by the optimized random features. We prove that the QML algorithm for sampling optimized random features, combined with stochastic gradient descent (SGD), can achieve state-of-the-art exponential convergence speed of reducing classification error in a classification task under a low-noise condition; at the same time, our algorithm with optimized random features can take advantage of the significant reduction of the required number of features so as to accelerate each iteration in the SGD and evaluation of the classifier obtained from our algorithm. These results discover a promising application of QML to significant acceleration of the leading classification algorithm based on kernel methods, without ruining its applicability to a practical class of data sets and the exponential error-convergence speed.Preprint. Under review.

show abstract

“…Another line of research has focused on data-dependent choice of random features. In [30,31,32,33], data-dependent random features has been studied for the approximation of shiftinvariant/translation-invariant kernels. On the other hand, in [34,35,36,37], the focal point is on the improvement of the out-of-sample error.…”

Section: Related Literaturementioning

confidence: 99%

Learning Bounds for Greedy Approximation with Explicit Feature Maps from Multiple Kernels

Shahrampour,

Tarokh

2018

Preprint

View full text Add to dashboard Cite

Nonlinear kernels can be approximated using finite-dimensional feature maps for efficient risk minimization. Due to the inherent trade-off between the dimension of the (mapped) feature space and the approximation accuracy, the key problem is to identify promising (explicit) features leading to a satisfactory out-of-sample performance. In this work, we tackle this problem by efficiently choosing such features from multiple kernels in a greedy fashion. Our method sequentially selects these explicit features from a set of candidate features using a correlation metric. We establish an out-of-sample error bound capturing the trade-off between the error in terms of explicit features (approximation error) and the error due to spectral properties of the best model in the Hilbert space associated to the combined kernel (spectral error). The result verifies that when the (best) underlying data model is sparse enough, i.e., the spectral error is negligible, one can control the test error with a small number of explicit features, that can scale poly-logarithmically with data. Our empirical results show that given a fixed number of explicit features, the method can achieve a lower test error with a smaller time cost, compared to the state-of-the-art in data-dependent random features.

show abstract

Data-driven Random Fourier Features using Stein Effect

Cited by 15 publications

References 1 publication

Random Features for Kernel Approximation: A Survey on Algorithms, Theory, and Beyond

Random Features for Kernel Approximation: A Survey on Algorithms, Theory, and Beyond

Exponential Error Convergence in Data Classification with Optimized Random Features: Acceleration by Quantum Machine Learning

Learning Bounds for Greedy Approximation with Explicit Feature Maps from Multiple Kernels

Contact Info

Product

Resources

About