TrajSpark: A Scalable and Efficient In-Memory Management System for Big Trajectory Data

Zhang, Zhigang; Jin, Cheqing; Mao, Jiali; Yang, Xiaolin; Zhou, Aoying

doi:10.1007/978-3-319-63579-8_2

Cited by 29 publications

(31 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Zhang et al [10] propose a distributed in-memory system TrajSpark, which offers efficient management for mass trajectory data. It uses IndexTRDD to provide efficient compression storage and parallel query support, and tracts changes in data distribution through the time decay model to continuously support efficient management over the increasing mass trajectory data.…”

Section: Related Workmentioning

confidence: 99%

“…While our scheme supports a bulk-based partitioning model and certain optimization strategies to alleviate the cost of repartitioning, so that the incremental dataset can be supported efficiently. (2) Most of above schemes [9][10][11] do not consider the effect of data transmission on queries, they distribute the sub-trajectories and index data of the same MO in different partitions on different machines, and additional across-node network overhead is required during the query to handle the merger of the index or trajectory data of the same MO. While our scheme stores the data of the same MO in a partition, co-resides each partition and its corresponding index on the same node, and implements node-locality-based parallel query algorithms to reduce the data transmission overhead, thereby improving query efficiency.…”

Section: Related Workmentioning

confidence: 99%

“…In order to process the similarity query problem of large-scale trajectory data, some distributed framework based schemes [9][10][11][12] are proposed according to the selected distance functions. However, most of them [9,11,12] only support the static datasets, while in many real-life trajectory applications, the data is incrementally appended, and these schemes are difficult to apply directly.…”

Section: Introductionmentioning

confidence: 99%

“…When each new batch of data arrives, these schemes should repartition all data from scratch, which leads to an expensive processing cost, and sacrifices flexibility and efficiency. Moreover, most of the above schemes [9][10][11] do not consider the effect of data transmission on queries, and they distribute the sub-trajectories and index data of the same MO in different partitions on different machines. In query processing, a large amount of across-node network overhead is required to handle the merger of the index or trajectory data.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

DFTHR: A Distributed Framework for Trajectory Similarity Query Based on HBase and Redis

Qin

Liu

2019

Information

View full text Add to dashboard Cite

In recent years positioning sensors have become ubiquitous, and there has been tremendous growth in the amount of trajectory data. It is a huge challenge to efficiently store and query massive trajectory data. Among the typical operation over trajectories, similarity query is an important yet complicated operator. It is useful in navigation systems, transportation optimizations, and so on. However, most existing studies have focused on handling the problem on a centralized system, while with a single machine it is difficult to satisfy the storage and processing requirements of mass data. A distributed framework for the similarity query of massive trajectory data is urgently needed. In this research, we propose DFTHR (distributed framework based on HBase and Redis) to support the similarity query using Hausdorff distance. DFTHR utilizes a segment-based data model with a number of optimizations for storing, indexing and pruning to ensure efficient querying capability. Furthermore, it adopts a bulk-based method to alleviate the cost for adjusting partitions, so that the incremental dataset can be efficiently supported. Additionally, DFTHR introduces a co-location-based distributed strategy and a node-locality-based parallel query algorithm to reduce the inter-worker cost overhead. Experiments show that DFTHR significantly outperforms other schemes.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

DFTHR: A Distributed Framework for Trajectory Similarity Query Based on HBase and Redis

Qin

Liu

2019

Information

View full text Add to dashboard Cite

show abstract

“…MD-HBase [31], R-HBase [14], and GeoMesa [15] use distributed key/value stores. TrajSpark [48] is an in-memory system based on Spark and equipped with a two-level spatio-temporal index. While we do not address distributed computing in this article, we remark that, in general, a compressed storage enables the distribution of spatiotemporal data across fewer computers, thereby reducing the communication costs.The closest predecessor of our GraCT index is SEST-Index [12,45].…”

mentioning

confidence: 99%

GraCT: A Grammar-based Compressed Index for Trajectory Data

Brisaboa

Gómez-Brandón

Navarro

et al. 2019

Information Sciences

View full text Add to dashboard Cite

We introduce a compressed data structure for the storage of free trajectories of moving objects (such as ships and planes) that efficiently supports various spatio-temporal queries. Our structure, dubbed GraCT, stores the absolute positions of all the objects at regular time intervals (snapshots) using a k 2 -tree, which is a space-and time-efficient version of a region quadtree. Positions between snapshots are represented as logs of relative movements and compressed using Re-Pair, a grammar-based compressor. The nonterminals of this grammar are enhanced with MBR information to enable fast queries.The GraCT structure of a dataset occupies less than the raw data compressed with a powerful traditional compressor such as p7zip. Further, instead of requiring full decompression to access the data like a traditional compressor, GraCT supports direct access to object trajectories or to their position at specific time instants, as well as spatial range and nearest-neighbor queries on time instants and/or time intervals.Compared to traditional methods for storing and indexing spatio-temporal data, GraCT requires two orders of magnitude less space, and is competitive in query times. In particular, thanks to its compressed representation, the GraCT structure may reside in main memory in situations where any classical uncompressed index must resort to disk, thereby being one or two orders of magnitude faster.Indexes that handle free trajectories of points in the space are also called spatiotemporal indexes. One type spatio-temporal index builds on a classic multidimensional spatial index, in the form of a temporally augmented R-tree [13]. While the R-tree uses two-dimensional Minimum Bounding Rectangles (MBRs) to enclose the spatial objects stored in the database, the 3DR-tree [42] uses Minimum Bounding Boxes (MBBs), where the third dimension is time, to enclose the segments of the trajectories. The problem is that the three-dimensional space covered by an MBB can be large, even in small segments, resulting in high levels of overlap and limited power of discrimination. The STR-Tree [32] is an extension of the 3DR-tree designed to overcome this problem, by modifying the insertion/split strategy that produces the MBBs. The same work also proposes an index called TB-tree, where several segments of a trajectory are bundled into partial trajectories that are inserted as MBBs of an R-tree.A second kind of index is a versioned R-tree, which creates an R-tree for each timestamp and a B-tree to select the relevant R-trees. Creating an R-tree for each timestamp requires large amounts of space. To overcome this, instead of storing the complete R-tree for each timestamp, these techniques store only the part of the R-tree that is different from the R-tree in the previous timestamp. Examples of indexes of this type are MR-tree [46], HR-tree [27], HR+-tree [39] and MV3R-tree [40]. A third family of indexes, called grid-based indexes, partition the space and build a temporal index for each of the spatial partitions. The Scalable and Efficie...

show abstract

Prospective Data Model and Distributed Query Processing for Mobile Sensing Data Streams

Brahem

Zeitouni

Yeh

et al. 2020

Lecture Notes in Computer Science

View full text Add to dashboard Cite

With the rapid advancements of sensor technologies and mobile computing, Mobile Crowd-Sensing (MCS) has emerged as a new paradigm to collect massive-scale rich trajectory data. Nomadic sensors empower people and objects with the capability of reporting and sharing observations on their state, their behavior and/or their surrounding environments. Processing and analyzing this continuously growing data raise several challenges due not only to their volume, their velocity, and their complexity but also to the gap between raw data samples and the desired application view in terms of correlation between observations and in terms of granularity. In this paper, we put forward a proposal that offers an abstract view of any spatio-temporal data series as well as their manipulation. Our approach allows to support this high-level logical view and provides efficient processing by mapping both the representation and the manipulation to an internal physical model. We explore an implementation within a distributed framework and envision the adaptation of data organization methods combining aggressive indexing and partitioning over time and space. The mapping from the logical view and the actual data storage will lead to revisiting the traditional database query rewriting and optimization techniques. This proposal is a first step in the objective of coping with the complexity, the imperfection of large data sizes in the MCS context.

show abstract

TrajSpark: A Scalable and Efficient In-Memory Management System for Big Trajectory Data

Cited by 29 publications

References 15 publications

DFTHR: A Distributed Framework for Trajectory Similarity Query Based on HBase and Redis

DFTHR: A Distributed Framework for Trajectory Similarity Query Based on HBase and Redis

GraCT: A Grammar-based Compressed Index for Trajectory Data

Prospective Data Model and Distributed Query Processing for Mobile Sensing Data Streams

Contact Info

Product

Resources

About