Classification in general finite dimensional spaces with the k-nearest neighbor rule

Optimal functional supervised classification with separation condition

“…We derive an excess risk bound that generalizes the worst-case results of [7] but also imply faster rates when the distance f − g is large. This acceleration is a consequence of the nice properties of the margin (see also, e.g., [2] and [14]).…”

“…This setting has been extensively studied so far. Popular classification procedures that are now theoretically well understood include the ERM method [27,2], the k-nearest neighbors algorithm [16,9,3,14], support vector machines [32], or random forests [4], just to name a few.…”

“…These bounds are parametrized by a separation distance quantifying the difficulty of the problem, and are proved to be optimal (up to logarithmic factors) through matching minimax lower bounds. Using the recent works of [9] and [14] we also derive a logarithmic lower bound showing that the popular k-nearest neighbors classifier is far from optimality in this specific functional setting.Theorem (A). The classifier Φ dn defined in (16) with dn ≈ n 1 2s+1 has an excess risk roughly bounded by (omitting logarithmic factors and constant factors depending only on s and R): for n NR,s large enough,• In Section 4.1 we derive a matching minimax lower bound (up to logarithmic factors) showing that the above worst-case bound cannot be improved by any classifier.Theorem (B).…”

“…These bounds are parametrized by a separation distance quantifying the difficulty of the problem, and are proved to be optimal (up to logarithmic factors) through matching minimax lower bounds. Using the recent works of [9] and [14] we also derive a logarithmic lower bound showing that the popular k-nearest neighbors classifier is far from optimality in this specific functional setting.…”

Explaining the Success of Nearest Neighbor Methods in Prediction

“…We present theoretical guarantees for k-nearest neighbor, fixed-radius near neighbor, and kernel regression where the data reside in a metric space. The proofs borrow heavily from the work by Chaudhuri and Dasgupta (2014) with some influence from the work by Gadat et al (2016). These authors actually focus on classification, but proof ideas translate over to the regression setting.…”