Efficient Group K Nearest-Neighbor Spatial Query Processing in Apache Spark

Abstract

Aiming at the problem of spatial query processing in distributed computing systems, the design and implementation of new distributed spatial query algorithms is a current challenge. Apache Spark is a memory-based framework suitable for real-time and batch processing. Spark-based systems allow users to work on distributed in-memory data, without worrying about the data distribution mechanism and fault-tolerance. Given two datasets of points (called Query and Training), the group K nearest-neighbor (GKNN) query retrieves (K) points of the Training with the smallest sum of distances to every point of the Query. This spatial query has been actively studied in centralized environments and several performance improving techniques and pruning heuristics have been also proposed, while, a distributed algorithm in Apache Hadoop was recently proposed by our team. Since, in general, Apache Hadoop exhibits lower performance than Spark, in this paper, we present the first distributed GKNN query algorithm in Apache Spark and compare it against the one in Apache Hadoop. This algorithm incorporates programming features and facilities that are specific to Apache Spark. Moreover, techniques that improve performance and are applicable in Apache Spark are also incorporated. The results of an extensive set of experiments with real-world spatial datasets are presented, demonstrating that our Apache Spark GKNN solution, with its improvements, is efficient and a clear winner in comparison to processing this query in Apache Hadoop.

A Novel Query Method for Spatial Database Based on Improved K-Nearest Neighbor Algorithm

“…Voronoi diagram (VD) plays the role as a very effective tool in computing geometries. In recent years, VDs have been widely used in spatial databases to describe spatial neighbor relationships; they also are used for realization of a spatial neighbor query, spatial interpolation, and buffer analysis (Moutafis et al, 2021).…”

A PID-Based kNN Query Processing Algorithm for Spatial Data

“…Because Hadoop MapReduce is a disk-based distributed computing framework, its response time is relatively slow, so it is not suitable for online queries. Therefore, more and more researchers have proposed spatial query processing algorithms based on the Spark framework [ 4 , 5 , 6 , 7 , 24 , 25 , 26 , 27 , 28 ], which is an in-memory computing framework with a faster processing speed. Xie [ 6 ] proposed a spatial big data analysis system based on Spark, which expanded the Spark SQL engine and supported the construction of an RDD-based memory index, thus effectively supporting various query operations such as range query and kNN query.…”

“…Compared with the original Spark, SparkNN significantly improves the average query time. Moutafis [ 26 ] proposed the first distributed GKNN query algorithm in Apache Spark, and this method proved to be more efficient than Apache Hadoop.…”