K Nearest Neighbour Joins for Big Data on MapReduce: A Theoretical and Experimental Analysis

Abstract

International audienceGiven a point p and a set of points S, the kNN operation finds the k closest points to p in S. It is a computational intensive task with a large range of applications such as knowledge discovery or data mining. However, as the volume and the dimension of data increase, only distributed approaches can perform such costly operation in a reasonable time. Recent works have focused on implementing efficient solutions using the MapReduce programming model because it is suitable for distributed large scale data processing. Although these works provide different solutions to the same problem, each one has particular constraints and properties. In this paper, we compare the different existing approaches for computing kNN on MapReduce, first theoretically, and then by performing an extensive experimental evaluation. To be able to compare solutions, we identify three generic steps for kNN computation on MapReduce: data pre-processing, data partitioning and computation. We then analyze each step from load balancing, accuracy and complexity aspects. Experiments in this paper use a variety of datasets, and analyze the impact of data volume, data dimension and the value of k from many perspectives like time and space complexity, and accuracy. The experimental part brings new advantages and shortcomings that are discussed for each algorithm. To the best of our knowledge, this is the first paper that compares kNN computing methods on MapReduce both theoretically and experimentally with the same setting. Overall, this paper can be used as a guide to tackle kNN-based practical problems in the context of big data

Constrained Oversampling: An Oversampling Approach to Reduce Noise Generation in Imbalanced Datasets With Class Overlapping

“…In the first step of our method, we build a KNN-based method to extract overlapping regions included in the original dataset. As a geometric classifier, the KNN classifier determines the class membership of a data point from its distance to reference data points [39], and it is widely used in clustering [40], [41], [42]. Thus, KNN is a density-based method to classify the input data by the percentage of samples belonging to a different class in their nearest neighbors.…”

Cooperative Bargain for the Autonomous Separation of Traffic Flows in Smart Reversible Lanes

“…Recently, much attention has been paid to the cooperative broadcast, where the message recovery is offloaded to the nodes in a cooperative manner [22,23]. For instance, a cooperative beacon broadcast scheme is proposed in [24] to provide the vehicles with more traffic information when driving on roads.…”

Data Distribution Aware Classification Algorithm based on K-Means

“…Authors of [14] compare and analyse five different existing methods to deduce the strengths and weaknesses of the KNN classification scheme for big data. As evaluation, [14] presents the advantages and disadvantages of the different stages of the compared classification models which are all applied on MapReduce work-flow.…”

“…As evaluation, [14] presents the advantages and disadvantages of the different stages of the compared classification models which are all applied on MapReduce work-flow. It is claimed in [14] that the results achieved in the study can be used to tackle different practical KNN problems in the context of big data.…”