Spinner: Scalable Graph Partitioning in the Cloud

Abstract

Several organizations, like social networks, store and routinely analyze large graphs as part of their daily operation. Such graphs are typically distributed across multiple servers, and graph partitioning is critical for efficient graph management. Existing partitioning algorithms focus on finding graph partitions with good locality, but disregard the pragmatic challenges of integrating partitioning into large-scale graph management systems deployed on the cloud, such as dealing with the scale and dynamicity of the graph and the compute environment.In this paper, we propose Spinner, a scalable and adaptive graph partitioning algorithm based on label propagation designed on top of the Pregel model. Spinner scales to massive graphs, produces partitions with locality and balance comparable to the state-of-the-art and efficiently adapts the partitioning upon changes. We describe our algorithm and its implementation in the Pregel programming model that makes it possible to partition billion-vertex graphs. We evaluate Spinner with a variety of synthetic and real graphs and show that it can compute partitions with quality comparable to the state-of-the art. In fact, by using Spinner in conjunction with the Giraph graph processing engine, we speed up different applications by a factor of 2 relative to standard hash partitioning.

Multi-dimensional balanced graph partitioning via projected gradient descent

“…Our initial experiments (see Figure 1) and earlier works [18,29,33] indicate that two important dimensions for the performance of Giraph jobs are the number of vertices and the number of edges. For this reason, we specify two weights for the vertices, w…”

“…While one-dimensional balanced graph partitioning has been studied extensively and a number of tools exist [7,13,14,22,23,33,41,42] (see also surveys by Bichot and Siarry [9] and by Buluç et al [12]), to the best of our knowledge none of the practical algorithms for this problem have been previously based on running gradient descent on a continuous relaxation. Existing approaches are inherently discrete and are based on combinations of various discrete algorithms: greedy heuristics (METIS [23], Fennel [41]), branch-and-bound [13], label propagation and local search (balanced label propagation [42], Social Hash Partitioner [22], Spinner [33]), as well as hybrid approaches (linear embedding method combined with various optimizations [7]). Due to the combinatorial nature of these algorithms, their generalizations to the multi-dimensional case appear to be non-straightforward without substantial losses in performance, while our continuous relaxation handles multiple balance constraints uniformly.…”

HYPE: Massive Hypergraph Partitioning with Neighborhood Expansion

“…Multilevel graph partitioners such as METIS [29] and ParMETIS [30] do not scale very well. Spinner [31] is a highly scalable graph partitioner that, like SHP, performs iterative random permutations and greedy selection of the best permutation. There is a large number of streaming graph partitioning algorithms, such as HDRF [17], H-load [32], and ADWISE [33].…”

Q-graph

“…Several query-agnostic partitioning algorithms optimizing the number of cut vertices or edges have been proposed [7,15,23,25,30,34,36,37,42]. As shown in Section 4, even best-case edgecut partitioning algorithms lead to suboptimal locality, workload balancing, and query latency.…”