Scaling queries over big RDF graphs with semantic hash partitioning

Abstract

Massive volumes of big RDF data are growing beyond the performance capacity of conventional RDF data management systems operating on a single node. Applications using large RDF data demand efficient data partitioning solutions for supporting RDF data access on a cluster of compute nodes. In this paper we present a novel semantic hash partitioning approach and implement a Semantic HAsh Partitioning-Enabled distributed RDF data management system, called Shape. This paper makes three original contributions. First, the semantic hash partitioning approach we propose extends the simple hash partitioning method through direction-based triple groups and direction-based triple replications. The latter enhances the former by controlled data replication through intelligent utilization of data access locality, such that queries over big RDF graphs can be processed with zero or very small amount of inter-machine communication cost. Second, we generate locality-optimized query execution plans that are more efficient than popular multi-node RDF data management systems by effectively minimizing the inter-machine communication cost for query processing. Third but not the least, we provide a suite of locality-aware optimization techniques to further reduce the partition size and cut down on the inter-machine communication cost during distributed query processing. Experimental results show that our system scales well and can process big RDF datasets more efficiently than existing approaches.

Selectivity Estimation for SPARQL Triple Patterns with Shape Expressions

“…Most existing works are based on scanning the data a-priori and either saving new pieces of information about it, or providing alternative data representations. The works in [7,9,13,14,16] are based on techniques that mainly focus on join optimizations by indexing the data. These works do not consider structured data and data typing.…”

An Association-Oriented Partitioning Approach for Streaming Graph Query

“…Lee and Liu present a novel semantic hash approach that utilizes access locality to partition big graphs across multiple computing nodes by maximizing the intrapartition processing capability and minimizing the interpartition communication cost [22]. Huang et al use a graph partitioning algorithm instead of simple hash partitioning by source vertex, destination vertex, and labeled or unlabeled edge [23].…”

A Configuration Management Study to Fast Massive Writing for Distributed NoSQL System

“…And one more interesting thing is that the key range generated by YCSB is uniform, when turn to highly skewed key range distribution, a more carefully PreSplit design is critical to achieve load balance. In HConfig system, we allow external data partitioning algorithms such as [32] to be plugged into the PreSplit policy.…”