Streaming graph partitioning for large distributed graphs

Abstract

Extracting knowledge by performing computations on graphs is becoming increasingly challenging as graphs grow in size. A standard approach distributes the graph over a cluster of nodes, but performing computations on a distributed graph is expensive if large amount of data have to be moved. Without partitioning the graph, communication quickly becomes a limiting factor in scaling the system up. Existing graph partitioning heuristics incur high computation and communication cost on large graphs, sometimes as high as the future computation itself. Observing that the graph has to be loaded into the cluster, we ask if the partitioning can be done at the same time with a lightweight streaming algorithm.We propose natural, simple heuristics and compare their performance to hashing and METIS, a fast, offline heuristic. We show on a large collection of graph datasets that our heuristics are a significant improvement, with the best obtaining an average gain of 76%. The heuristics are scalable in the size of the graphs and the number of partitions. Using our streaming partitioning methods, we are able to speed up PageRank computations on Spark [32], a distributed computation system, by 18% to 39% for large social networks.

Efficient model partitioning for distributed model transformations

“…If the splits have similar score, then the element is assigned to the split with the lowest index. While different penalty functions can be used, we go for a linear weighted function, as it has experimentally shown good performance results [21]. This forces our distribution to be uniform.…”

“…Stanton and Kliot [21] compared a set of lightweight Streaming Graph Partitioning for Large Distributed Graphs and compare their performance to some well-known offline algorithms. They run their benchmark on large collections of datasets, and showed up to 76% of average gain.…”

“…Several task-parallel [5] and data-parallel [21] distribution techniques have been proposed, but none of them addresses the specific case of relational model transformation. In this paper we argue that when a MT is defined in a relational language, an efficient data-distribution 2 scheme can be derived by statically analysing the structure of the transformation rules.…”

Adaptive Partitioning for Large-Scale Dynamic Graphs

“…First, ensuring good application performance, our ultimate goal, requires graph partitions that minimise communication and achieve load balancing. Minimising communication allows more parallelisation and improves system scalability, while load balancing can have a significant impact on overall processing time [21,35]. However, these are often conflicting requirements.…”

Giraphx: Parallel Yet Serializable Large-Scale Graph Processing

“…A bandwidth aware graph partitioning framework to minimize the network traffic in partitioning and processing is proposed in [19]. Finally, another recent work [20] shows that using simple partitioning heuristics can bring a significant performance improvement that surpasses the widely-used offline metis partitioner.…”