Distribution-free uncertainty quantification for kernel methods by gradient perturbations

Abstract

We propose a data-driven approach to quantify the uncertainty of models constructed by kernel methods. Our approach minimizes the needed distributional assumptions, hence, instead of working with, for example, Gaussian processes or exponential families, it only requires knowledge about some mild regularity of the measurement noise, such as it is being symmetric or exchangeable. We show, by building on recent results from finite-sample system identification, that by perturbing the residuals in the gradient of the objective function, information can be extracted about the amount of uncertainty our model has. Particularly, we provide an algorithm to build exact, non-asymptotically guaranteed, distribution-free confidence regions for ideal, noise-free representations of the function we try to estimate. For the typical convex quadratic problems and symmetric noises, the regions are star convex centered around a given nominal estimate, and have efficient ellipsoidal outer approximations. Finally, we illustrate the ideas on typical kernel methods, such as LS-SVC, KRR, ε-SVR and kernelized LASSO.

A reinforcement learning algorithm for trading commodities

“…F I G U R E 1 WTI observed prices (A) and increments of the corresponding factor (B) computed as in (20).…”

“…In Figure 1A we display the WTI time series, in Figure 1B the corresponding factor time series computed as in (20). Maximum likelihood estimation of the GP model ( 12)-( 13) returns the parameters in Table 1 5 .…”

A review of uncertainty quantification in deep learning: Techniques, applications and challenges

“…Their method yielded well-calibrated predictive uncertainty and accurate predictions for both image classification and regression by exploiting Bayesian model averaging over the induced posterior in the subspaces. Csáji et al [242] introduced a data-driven strategy for uncertainty quantification of models based on kernel techniques. The method needed few mild regularities in the computation of noise instead of distributional assumptions such as dealing with exponential families or GPs.…”

Exact Distribution-Free Hypothesis Tests for the Regression Function of Binary Classification via Conditional Kernel Mean Embeddings

“…Motivated by finite-sample system identification methods [26], [30], [35], we also suggest two specific algorithms that build exact confidence regions for the regression function of binary classification, under mild statistical assumptions. These new methods are based on the concept of resampling, which is closely related to bootstrap and Monte Carlo tests.…”

“…It is also proved that SPS is strongly consistent [29] under mild statistical assumptions and comparable to the asymptotic confidence ellipsoids. SPS has been already generalized for kernel methods [30] and modified for classification [31], [32]. Nevertheless its analysis for classification is still narrow.…”