Distributed block formation and layout for disk-based management of large-scale graphs

Abstract

We are witnessing an enormous growth in social networks as well as in the volume of data generated by them. An important portion of this data is in the form of graphs. In recent years, several graph processing and management systems emerged to handle large-scale graphs. The primary goal of these systems is to run graph algorithms and queries in an efficient and scalable manner. Unlike relational data, graphs are semi-structured in nature. Thus, storing and accessing graph data using secondary storage requires new solutions that can provide locality of access for graph processing workloads. In this work, we propose a scalable block formation and layout technique for graphs, which aims at reducing the I/O cost of disk-based graph processing algorithms. To achieve this, we designed a scalable MapReducestyle method called ICBL, which can divide the graph into a series of disk blocks that contain sub-graphs with high locality. Furthermore, ICBL can order the resulting blocks on disk to further reduce non-local accesses. We experimentally evaluated ICBL to showcase its scalability, layout quality, as well as the effectiveness of automatic parameter tuning for ICBL. We deployed the graph layouts generated by ICBL on the Neo4j open source graph database, http://www.neo4j.org/ (2015) graph database management system. Our results show that the layout generated by ICBL reduces the query running times over Neo4j more than 2× compared to the default layout. B Abdurrahman Yaşar

mcatCS: A Highly Efficient Cross-matching Scheme for Multi-band Astronomical Catalogs

“…They constructed an undirected correlation graph and used the soft coloring algorithm to color adjacent nodes in different colors and the traditional binpacking algorithm to handle disk capacities. Yasar et al (2017) designed a distributed data layout technique for graphs; they assigned a rank label to each block and ordered them based on rank, which aims at reducing the I/O cost. In addition, heuristic algorithms, such as the genetic algorithm (e.g., Zhao et al 2013;Fan et al 2016), and particle swarm optimization (Wang et al 2014), have also been applied to data layout problems.…”

Cascade-aware partitioning of large graph databases

“…Distributed graph databases employ partitioning methods to provide data locality for queries and to keep the load balanced among servers [1][2][3][4][5]. Online social networks (OSNs) are common applications of graph databases where users are represented by vertices and their connections are represented by edges/hyperedges.…”