Hybrid Edge Partitioner: Partitioning Large Power-Law Graphs under Memory Constraints

Abstract

Distributed systems that manage and process graph-structured data internally solve a graph partitioning problem to minimize their communication overhead and query run-time. Besides computational complexity-optimal graph partitioning is NP-hard-another important consideration is the memory overhead. Real-world graphs often have an immense size, such that loading the complete graph into memory for partitioning is not economical or feasible. Currently, the common approach to reduce memory overhead is to rely on streaming partitioning algorithms. While the latest streaming algorithms lead to reasonable partitioning quality on some graphs, they are still not completely competitive to in-memory partitioners. In this paper, we propose a new system, Hybrid Edge Partitioner (HEP), that can partition graphs that fit partly into memory while yielding a high partitioning quality. HEP can flexibly adapt its memory overhead by separating the edge set of the graph into two sub-sets. One sub-set is partitioned by NE++, a novel, efficient in-memory algorithm, while the other sub-set is partitioned by a streaming approach. Our evaluations on large real-world graphs show that in many cases, HEP outperforms both in-memory partitioning and streaming partitioning at the same time. Hence, HEP is an attractive alternative to existing solutions that cannot finetune their memory overheads. Finally, we show that using HEP, we achieve a significant speedup of distributed graph processing jobs on Spark/GraphX compared to state-of-the-art partitioning algorithms. CCS CONCEPTS• Information systems → Graph-based database models; • Theory of computation → Graph algorithms analysis.

Out-of-Core Edge Partitioning at Linear Run-Time

“…If u or v have no cluster yet, we create a new cluster and assign the vertex to it (lines [11][12][13][14][15]. Now, we compare the cluster volumes of the clusters of u and v. The vertex that is currently assigned to the cluster with the lower volume migrates to the neighboring cluster that has the higher volume (lines [16][17][18][19][20][21][22]. However, such migration is only allowed if the new volume of the larger cluster does not exceed a volume bound.…”

“…Table III) that represent a good mix of social networks and web graphs of different sizes. OK is notoriously difficult to partition; IT, TW and FR [31], [32] are used as baselines in many graph partitioning papers [14], [19], [10], [17], [18]. UK, GSH [33], [34], [35] and WDC [36] represent very large graphs.…”

“…(3) Hybrid Partitioning. We also compare ourselves to Hybrid Edge Partitioner (HEP) [19] which operates between inmemory and streaming partitioning using a tunable parameter τ . We set τ to different values (100, 10, 1) as in the original publication [19].…”

“…We also compare ourselves to Hybrid Edge Partitioner (HEP) [19] which operates between inmemory and streaming partitioning using a tunable parameter τ . We set τ to different values (100, 10, 1) as in the original publication [19]. With a high setting of τ = 100, HEP behaves more like an in-memory partitioner, while with a low setting of τ = 1, HEP behaves more like a streaming partitioner.…”

“…In-memory partitioners [11], [12], [13], [10], [14] load the complete graph into memory and then perform partitioning on it. In contrast to this, out-of-core partitioners [15], [16], [17], [18], [19] forgo materializing a representation of the complete graph in memory. Instead, they commonly ingest the graph as a stream of edges, i.e., edge by edge, and assign each edge of the stream to a partition.…”