FALKON: An Optimal Large Scale Kernel Method

Abstract

Kernel methods provide a principled way to perform non linear, nonparametric learning. They rely on solid functional analytic foundations and enjoy optimal statistical properties. However, at least in their basic form, they have limited applicability in large scale scenarios because of stringent computational requirements in terms of time and especially memory. In this paper, we take a substantial step in scaling up kernel methods, proposing FALKON, a novel algorithm that allows to efficiently process millions of points. FALKON is derived combining several algorithmic principles, namely stochastic subsampling, iterative solvers and preconditioning. Our theoretical analysis shows that optimal statistical accuracy is achieved requiring essentially O(n) memory and O(n √ n) time. An extensive experimental analysis on large scale datasets shows that, even with a single machine, FALKON outperforms previous state of the art solutions, which exploit parallel/distributed architectures.

An Extensible Benchmark Suite for Learning to Simulate Physical Systems

“…The resulting maps x → ρ( x, z l ) are random features, associated with a shallow neural network with 'frozen' weights. While further choices of kernel may be considered in the future, dot-product kernels have flexible approximation properties and are easily scalable [40].…”

Learning new physics efficiently with nonparametric methods

“…( 24) is typically cubic in time and quadratic in space with respect to the number of points, since it requires handling the kernel matrix K N N ∈ R N ×N with entries k γ (x i , x j ) (see Refs. [40,49] for further details). These costs prevent the application of basic solvers in large-scale setting, and some approximation is needed.…”

“…The core ideas from a theoretical and algorithmic viewpoint are developed in Ref. [49,57,59]. The problem of minimizing the regularized empirical risk in Eq.…”

Neural Fields as Learnable Kernels for 3D Reconstruction

“…State-of-the-art Kernel solvers in the literature (e.g. [52]) have scaled up to millions of points by leveraging techniques such as Nyström sampling. We plan to investigate how to leverage these approaches to handle larger inputs.…”