Spatial partitioning techniques in SpatialHadoop

Abstract

SpatialHadoop is an extended MapReduce framework that supports global indexing that spatial partitions the data across machines providing orders of magnitude speedup, compared to traditional Hadoop. In this paper, we describe seven alternative partitioning techniques and experimentally study their effect on the quality of the generated index and the performance of range and spatial join queries. We found that using a 1% sample is enough to produce high quality partitions. Also, we found that the total area of partitions is a reasonable measure of the quality of indexes when running spatial join. This study will assist researchers in choosing a good spatial partitioning technique in distributed environments. 1. INDEXING IN SPATIALHADOOP SpatialHadoop [2, 3] provides a generic indexing algorithm which was used to implement grid, R-tree, and R+-tree based partitioning. This paper extends our previous study by introducing four new partitioning techniques, Z-curve, Hilbert curve, Quad tree, and K-d tree, and experimentally evaluate all of the seven techniques. The partitioning phase of the indexing algorithm runs in three steps, where the first step is fixed and the last two steps are customized for each partitioning technique. The first step computes number of desired partitions n based on file size and HDFS block capacity which are both fixed for all partitioning techniques. The second step reads a random sample, with a sampling ratio ρ, from the input file and uses this sample to partition the space into n cells such that number of sample points in each cell is at most ⌊k/n⌋, where k is the sample size. The third step actually partitions the file by assigning each record to one or more cells. Boundary objects are handled using either the distribution or replication methods. The distribution method assigns an object to exactly one overlapping cell and the cell has to be expanded to enclose all contained records. The replication method avoids expanding cells by replicating each record to all overlapping cells but the query processor has to employ a duplicate avoidance technique to account for replicated records.

Enhancing SpatialHadoop with Closest Pair Queries

“…In the Language layer, SpatialHadoop adds a simple and expressive high level language for spatial data types and operations. In the Storage layer, SpatialHadoop adapts traditional spatial index structures as Grid, R-tree and R + -tree, to form a two-level spatial index [27]. SpatialHadoop enriches the MapReduce layer by new components to implement efficient and scalable spatial data processing.…”

Approach to Accelerating Dissolved Vector Buffer Generation in Distributed In-Memory Cluster Architecture

“…Multiple matching methods and multiple assignment methods are adopted to address these issues [18,38]. They work perfectly for the spatial query and join operations, despite the slightly increased overhead in the latter method due to the multiple duplications.…”

A Novel Spatio-Temporal Data Storage and Index Method for ARM-Based Hadoop Server

“…In order to make uniform distribution of the spatial data and ensure the node load balance, we improve the quadtree algorithm [11] and design a bet- ter partitioning strategy for spatio-temporal data, and we term this algorithm the QaDTree algorithm, which generates the global index using the improved quadtree algorithm to partition the spatio-temporal data and the local indexes using 3DR-tree index to manage the spatio-temporal data [12]. The tree non-leaf node of the global index contains four children, each of which represents a subspace area.…”