Local nearest neighbour classification with applications to semi-supervised learning

Abstract

We derive a new asymptotic expansion for the global excess risk of a local-k-nearest neighbour classifier, where the choice of k may depend upon the test point. This expansion elucidates conditions under which the dominant contribution to the excess risk comes from the decision boundary of the optimal Bayes classifier, but we also show that if these conditions are not satisfied, then the dominant contribution may arise from the tails of the marginal distribution of the features. Moreover, we prove that, provided the d-dimensional marginal distribution of the features has a finite ρth moment for some ρ > 4 (as well as other regularity conditions), a local choice of k can yield a rate of convergence of the excess risk of O(n −4/(d+4) ), where n is the sample size, whereas for the standard k-nearest neighbour classifier, our theory would require d ≥ 5 and ρ > 4d/(d − 4) finite moments to achieve this rate. These results motivate a new k-nearest neighbour classifier for semi-supervised learning problems, where the unlabelled data are used to obtain an estimate of the marginal feature density, and fewer neighbours are used for classification when this density estimate is small. Our worst-case rates are complemented by a minimax lower bound, which reveals that the local, semi-supervised k-nearest neighbour classifier attains the minimax optimal rate over our classes for the excess risk, up to a subpolynomial factor in n. These theoretical improvements over the standard k-nearest neighbour classifier are also illustrated through a simulation study.

Strongly universally consistent nonparametric regression and classification with privatised data

“…Our work here allows us to strengthen this result to the following theorem on strong universal consistency: The rates of convergence of the classification rule g n , over classes of datagenerating mechanisms satisying Hölder continuity and a strong density assumption, were established in [3], and were moreover shown to match a minimax lower bound. We remark that, even in the non-private case, the absence of the strong density assumption leads to slower rates of convergence [22,1,7].…”

Classification with imperfect training labels

“…We now show that, under further regularity conditions on the data distribution P and the noise mechanism, it is possible to give a more precise description of the asymptotic error properties of the corrupted knn classifier. Since our conditions on P , which are slight simplifications of those used in Cannings et al (2018) to analyse the uncorrupted knn classifier, are a little technical, we give an informal summary of them here, deferring formal statements of our assumptions A1-A4 to just before the proof of Theorem 3 in Section A.2.…”

Classification with unknown class-conditional label noise on non-compact feature spaces

“…The major advantage of this is that we are able to avoid the restrictive assumption of an upper bound on the -covering numbers (which would rule out non-compact domains of interest). An alternative approach to noncompact domains has been pursued by Cannings et al (2017). Whilst we follow Gadat et al (2016) in bounding the measure of the regions of the feature space where the density falls below a given value (see Assumption E), Cannings et al (2017) instead employ a moment assumption.…”

“…An alternative approach to noncompact domains has been pursued by Cannings et al (2017). Whilst we follow Gadat et al (2016) in bounding the measure of the regions of the feature space where the density falls below a given value (see Assumption E), Cannings et al (2017) instead employ a moment assumption. Note that whereas Cannings et al (2017) make use of an additional set of unlabelled data to locally tune the optimal value of k, our method is optimally adaptive without any additional data.…”