Uncertainty Quantification in Graph-Based Classification of High Dimensional Data

Abstract

Classification of high dimensional data finds wide-ranging applications. In many of these applications equipping the resulting classification with a measure of uncertainty may be as important as the classification itself. In this paper we introduce, develop algorithms for, and investigate the properties of, a variety of Bayesian models for the task of binary classification; via the posterior distribution on the classification labels, these methods automatically give measures of uncertainty. The methods are all based around the graph formulation of semi-supervised learning. We provide a unified framework which brings together a variety of methods which have been introduced in different communities within the mathematical sciences. We study probit classification [43], generalize the level-set method for Bayesian inverse problems [24] to the classification setting, and generalize the Ginzburg-Landau optimization-based classifier [5, 40] to a Bayesian setting; we also show that the probit and level set approaches are natural relaxations of the harmonic function approach introduced in [49]. We introduce efficient numerical methods, suited to large data-sets, for both MCMC-based sampling as well as gradient-based MAP estimation. Through numerical experiments we study classification accuracy and uncertainty quantification for our models; these experiments showcase a suite of datasets commonly used to evaluate graph-based semi-supervised learning algorithms.

On the Well-posedness of Bayesian Inverse Problems

“…Such problems appear in Bayesian probabilistic numerics, and other settings where noise-free data is considered. This may also include the Bayesian formulation of machine learning problems with discrete loss models, like 0-1-loss, or Bayesian formulations of classification problems, see [4].…”

Ensemble Kalman inversion: a derivative-free technique for machine learning tasks

“…There are a variety of ways in which one can construct the regularizer R(u; x) including graph-based and low-density separation methods [7,6]. In this work, we will study a nonparametric graph approach where we think of Z as indexing the nodes on a graph.…”

“…For supervised and online learning these amount to specifying the dependence of G on u; for semi-supervised learning this corresponds to determining a basis in which to seek the parameter u. We do not give further details for the semi-supervised case as our numerics fit in the context of Example 2.2, but we refer the reader to [7] for a detailed discussion. Subsection 3.1 details feed-forward neural networks with subsections 3.1.1 and 3.1.2 showing the parameterizations of dense and convolutional networks respectively.…”

Properly-Weighted Graph Laplacian for Semi-supervised Learning

“…Other approaches include higher order Laplacian regularization [6,17,53] and using a spectral cut-off [5].…”

“…To illustrate what is happening near a labeled point, consider Γ " t0u and take the domain from which the points are sampled to be the unit ball Ω " Bp0, 1q in R d . The continuum variational problem corresponding to (5) involves minimizing (6) Irus " ż Bp0,1q…”