Boosting Nearest Neighbors for the Efficient Estimation of Posteriors

Abstract

It is an admitted fact that mainstream boosting algorithms like AdaBoost do not perform well to estimate class conditional probabilities. In this paper, we analyze, in the light of this problem, a recent algorithm, unn, which leverages nearest neighbors while minimizing a convex loss. Our contribution is threefold. First, we show that there exists a subclass of surrogate losses, elsewhere called balanced, whose minimization brings simple and statistically efficient estimators for Bayes posteriors. Second, we show explicit convergence rates towards these estimators for unn, for any such surrogate loss, under a Weak Learning Assumption which parallels that of classical boosting results. Third and last, we provide experiments and comparisons on synthetic and real datasets, including the challenging SUN computer vision database. Results clearly display that boosting nearest neighbors may provide highly accurate estimators, sometimes more than a hundred times more accurate than those of other contenders like support vector machines.

Gentle Nearest Neighbors Boosting over Proper Scoring Rules

“…Suppose we pick δ e(t) as in (10), that is, δ e(t) = 2(1−ε)η(c, e(t))(H * ψ φ n e(t) ) −1 , for ε ∈ (0, 1). This choice yields:…”

“…A previous approach in our line of works is algorithm UNN (for "Universal Nearest Neighbors"), which brings boosting guarantees for merely all strictly convex differentiable surrogates relevant to classification [9], [5], [6]. For a wide subset of surrogates, it yields simple and efficient estimators of posteriors [10].…”

Biomedical Images Classification by Universal Nearest Neighbours Classifier Using Posterior Probability

“…We report estimator's formal definition for each loss function of unn in Table 2. The theorical approach for derivingp c (x) from h c (x) is fully given in [5]. We used one private and five public datasets, belonging to images classification problems of different biomedical domains.…”

“…Reliabilities of unn implementations tested measured in terms of posterior probabilitiesp c (x) are computed as reported in Table 2. For further details, the interested reader may refer to [5]. To estimate the posterior probabilities for SVM we use the method presented in [13].…”

“…Then, a reconstruction rule provides the final classification [2][3][4]. Furthermore, it was proven that unn classifier can effectively estimates the posterior probability of each classification act [5]. This permits us to use this information to apply reconstruction rules potentially more effective than others criteria proposed in the literature, which set the final decision using the crisp labels provided by the binary learners.…”