Local nearest neighbour classification with applications to semi-supervised learning

Abstract

We derive a new asymptotic expansion for the global excess risk of a local-k-nearest neighbour classifier, where the choice of k may depend upon the test point. This expansion elucidates conditions under which the dominant contribution to the excess risk comes from the decision boundary of the optimal Bayes classifier, but we also show that if these conditions are not satisfied, then the dominant contribution may arise from the tails of the marginal distribution of the features. Moreover, we prove that, provided the d-dimensional marginal distribution of the features has a finite ρth moment for some ρ > 4 (as well as other regularity conditions), a local choice of k can yield a rate of convergence of the excess risk of O(n −4/(d+4) ), where n is the sample size, whereas for the standard k-nearest neighbour classifier, our theory would require d ≥ 5 and ρ > 4d/(d − 4) finite moments to achieve this rate. These results motivate a new k-nearest neighbour classifier for semi-supervised learning problems, where the unlabelled data are used to obtain an estimate of the marginal feature density, and fewer neighbours are used for classification when this density estimate is small. Our worst-case rates are complemented by a minimax lower bound, which reveals that the local, semi-supervised k-nearest neighbour classifier attains the minimax optimal rate over our classes for the excess risk, up to a subpolynomial factor in n. These theoretical improvements over the standard k-nearest neighbour classifier are also illustrated through a simulation study.

Minimax Rate Optimal Adaptive Nearest Neighbor Classification and Regression

“…The purpose of adding 1 to Kn q in (13) is to ensure that k is at least 1. Our method shares some similarity with [19], which uses the result of kernel density estimate to determine k. However, [19] requires a sufficiently large number of unlabeled data to ensure that the estimated density function is sufficiently close to the real density function, so that the adaptive kNN algorithm converges as fast as the case in which f (x) is known and the selection of k is based on the real f (x). On the contrary, our method does not require unlabeled data, and we do not hope to have an accurate estimation of the density.…”

“…Here, we use (17) to replace the Hölder condition, so that it is possible to impose an assumption that is approximately the same as requiring the second-order smoothness of η. Assumption 1 (d) is the minimum probability mass assumption, which was already used in existing works [18,19].…”

“…If apart from a set of labeled training samples, we also have abundant unlabeled samples, which are many more than labeled samples, then it is possible to estimate the pdf with sufficient accuracy. We can then use the estimated density values to determine the optimal k. In this case, the problem is actually a semi-supervised learning problem, which was discussed in [19]. However, in supervised learning problems, those unlabeled data is not available.…”

High-dimensional semi-supervised learning: in search for optimal inference of the mean

“…In the context of binary classification, estimation properties through various forms of dimension reduction have been studied by, among others Chapelle et al (2009); Grandvalet and Bengio (2005); Zhu (2005); Zhu and Goldberg (2009). For example, Cannings et al (2017) recently proposed a local k-nearest neighbors which uses fewer neighbors by utilizing the additional data to estimate the marginal density of features. They provide asymptotic results for the excess risk over R p , O(n −4/(p+4) ) when E X 4+c 2 < ∞ and n 2+p/r = O(m), where c > 0, r ∈ (0, 2] is the smoothness of the covariates' density.…”

Predictive Power of Nearest Neighbors Algorithm under Random Perturbation

“…Second, although the regret of k-NN deteriorates polynomially with respect to the corruption level, it actually achieves the best possible accuracy for testing randomly perturbed data (under a fine tuned choice of k). Hence k-NN classifier is rate-minimax for both clean data testing task [2,22,4] and randomly perturbed data testing task.…”