Nearest neighbor regression in the presence of bad hubs

Abstract

Prediction on a numeric scale, i.e., regression, is one of the most prominent machine learning tasks with various applications in finance, medicine, social and natural sciences. Due to its simplicity, theoretical performance guarantees and successful real-world applications, one of the most popular regression techniques is the k nearest neighbor regression. However, k nearest neighbor approaches are affected by the presence of bad hubs, a recently observed phenomenon according to which some of the instances are similar to surprisingly many other instances and have a detrimental effect on the overall prediction performance. This paper is the first to study bad hubs in context of regression. We propose hubness-aware nearest neighbor regression schemes. We evaluate our approaches on publicly available real-world datasets from various domains. Our results show that the proposed approaches outperform various other regressions schemes such as kNN regression, regression trees and neural networks. We also evaluate the proposed approaches in the presence of label noise because tolerance to noise is one of the most relevant aspects from the point of view of real-world applications. In particular, we perform experiments under the assumption of conventional Gaussian label noise and an adapted version of the recently proposed hubness-proportional random label noise.

Drug–target interaction prediction with Bipartite Local Models and hubness-aware regression

“…This means, roughly speaking, that bad hubs are expected in complex data, such as drug-target interaction data. For a more detailed discussion, we refer to [7].…”

“…[8] [28] [46], and hubness-aware classifiers have been developed, see [45] for a survey. More recently, hubness-aware regression techniques, including knearest neighbor with error correction (ECkNN), were developed that allow for predictions on a continuous scale [7]. Despite the fact that hubness-aware techniques are among the most promising recent machine learning approaches, their potential to enhance drug-target interaction prediction methods has not been exploited yet: to the best of our knowledge, our initial work [6] is the only one aiming to apply hubness-aware models to the drug-target prediction problem.…”

A New Proposal for Person Identification Based on the Dynamics of Typing: Preliminary Results

“…In order to keep the example simple, we use k = 1 nearest neighbour to calculate the corrected labels of training instances. In the figure, directed edges point from each instance to its first nearest For more details about ECkNN we refer to [18]. As mentioned previously, the dynamics of typing is captured by time series data.…”

“…We set k = 5 for ECkNN which is in accordance with other works on hubness-aware machine learning [18,22].…”

A comprehensive empirical comparison of hubness reduction in high-dimensional spaces

“…Hubness has been identified as a detrimental factor in similarity-based machine learning, impairing several classification [44], clustering [47,60], regression [7], graph analysis [22], visualization [17], and outlier detection [18,19,45] methods. Reports on affected tasks include multimedia retrieval [51], recommendation [48], collaborative filtering [25,34], speaker verification [50], speech recognition [62], and image data classification [58].…”