Enhancing SpatialHadoop with Closest Pair Queries

Abstract

Abstract. Given two datasets P and Q, the K Closest Pair Query (KCPQ) finds the K closest pairs of objects from P ×Q. It is an operation widely adopted by many spatial and GIS applications. As a combination of the K Nearest Neighbor (KNN) and the spatial join queries, KCPQ is an expensive operation. Given the increasing volume of spatial data, it is difficult to perform a KCPQ on a centralized machine efficiently. For this reason, this paper addresses the problem of computing the KCPQ on big spatial datasets in SpatialHadoop, an extension of Hadoop that supports spatial operations efficiently, and proposes a novel algorithm in SpatialHadoop to perform efficient parallel KCPQ on large-scale spatial datasets. We have evaluated the performance of the algorithm in several situations with big synthetic and real-world datasets. The experiments have demonstrated the efficiency and scalability of our proposal.

Efficient large-scale distance-based join queries in spatialhadoop

“…This paper substantially extends our previous work [1], which was the foundation of the present research results, with the following novel contributions:…”

“…We improve the plane-sweep-based KCPQ MapReduce algorithm in SpatialHadoop [1] by using new sampling and approximate techniques, that take advantage of SpatialHadoop partitioning techniques, to compute an upper bound of the distance of the K-th closest pair and make the KCQP MapReduce algorithm much more efficient. 2.…”

“…We present results of an extensive experimental study that compares the performance of the proposed MapReduce algorithms and their improvements in terms of efficiency and scalability. For synthetic datasets' experiments, we have used clustered (more realistic) datasets, instead of uniform ones [1]. More-over, for real datasets' experiments, we have created a new big quasi-real dataset that is combined with the biggest real dataset used in [1].…”

“…4. In experiments of DBJQ MapReduce algorithms, we utilize additional partitioning techniques available in SpatialHadoop to check if performance improvements are obtained with respect to the partitioning used in [1]. 5.…”

“…A number of parallel algorithms for DBJQs, like the K Closest Pair Query (KCPQ) [1], K Nearest Neighbor Join (KNNJ) [12][13][14][15] and similarity join [16] in MapReduce [17] have been designed and implemented recently. However, as real-world spatial datasets continue to grow, novel approaches and paradigms are needed.…”