Online nonparametric regression with Sobolev kernels

Abstract

In this work we investigate the variation of the online kernelized ridge regression algorithm in the setting of d−dimensional adversarial nonparametric regression. We derive the regret upper bounds on the classes of Sobolev spaces W β p (X ), p ≥ 2, β > d p . The upper bounds are supported by the minimax regret analysis, which reveals that in the cases β > d 2 or p = ∞ these rates are (essentially) optimal. Finally, we compare the performance of the kernelized ridge regression forecaster to the known non-parametric forecasters in terms of the regret rates and their computational complexity as well as to the excess risk rates in the setting of statistical (i.i.d.) nonparametric regression.

Scale-free Unconstrained Online Learning for Curved Losses

“…Comparator-Adaptive Regression over Sobolev Spaces Thanks to Zadorozhnyi et al (2021, Theorem 3), the results above imply adaptive rates when the class of functions is the Sobolev space W s,p ([−1, 1] d ) with p 2; we refer the reader to Adams and Fournier (2003) for definitions and properties of Sobolev spaces, and to Wendland (2004, Chapter 10) for more details on Sobolev spaces as RKHS. For simplicity, let us state the results in the case when s is an integer and s d/2.…”

“…Aggregated-KAAR A key algorithm in online regression is the Azoury-Vovk-Warmuth forecaster (also called the forward algorithm) from Azoury and Warmuth (2001); Vovk (1998), and its kernelised version KAAR (Gammerman et al, 2004). This algorithm has been analysed and modified in a variety of settings; see, e.g., Orabona et al (2015); Jézéquel et al (2019); ; Jézéquel et al (2019); Zadorozhnyi et al (2021). Upon seeing x t , KAAR with regularisation λ > 0 predicts a t = θ t (x t ) where θ t ∈ F is picked according to the rule…”