Optimizing graph algorithms on pregel-like systems

Abstract

We study the problem of implementing graph algorithms efficiently on Pregel-like systems, which can be surprisingly challenging. Standard graph algorithms in this setting can incur unnecessary inefficiencies such as slow convergence or high communication or computation cost, typically due to structural properties of the input graphs such as large diameters or skew in component sizes. We describe several optimization techniques to address these inefficiencies. Our most general technique is based on the idea of performing some serial computation on a tiny fraction of the input graph, complementing Pregel's vertex-centric parallelism. We base our study on thorough implementations of several fundamental graph algorithms, some of which have, to the best of our knowledge, not been implemented on Pregel-like systems before. The algorithms and optimizations we describe are fully implemented in our open-source Pregel implementation. We present detailed experiments showing that our optimization techniques improve runtime significantly on a variety of very large graph datasets.

Pregel algorithms for graph connectivity problems with performance guarantees

“…The first algorithm propagates the smallest vertex (ID) that every vertex has seen so far, while the second algorithm propagates multiple source vertices to speed up SCC computation. We also noticed a very recent algorithm that computes SCCs in Pregel [17], which shares a similar idea as our first algorithm. However, their algorithm performs label propagation for only one round, followed by a serial computation by the master machine, which is called FCS (Finishing Computations Serially).…”

“…Besides this paper and Google's original paper on Pregel [12], we are only aware of two other papers studying Pregel algorithms, [13] and [17]. However, the algorithms are designed on a best-effort basis without any formal analysis on their complexity.…”

“…However, the algorithms are designed on a best-effort basis without any formal analysis on their complexity. Specifically, [13] aims at demonstrating that the Pregel model can be adopted to solve many graph problems in social network analysis, while [17] focuses on optimization techniques that overcome some performance bottlenecks caused by straightforward Pregel implementations.…”

An experimental comparison of pregel-like graph processing systems

“…This tree is unique if all edge weights are distinct. We use the parallel Boruvka algorithm [11,33]. The input graph must be undirected but need not be connected.…”

Large-Scale Graph Processing Using Apache Giraph

“…Since Giraph's computing model is directed and distributed; it does not naturally support undirected graphs or have the ability to synchronize value assignment to adjacent vertices. To solve this problem, researchers tend to modify Pregel API to serialize the value assignment process through the master compute, as suggested by the work in [48,49]. In this section, we show how to handle undirected graphs and discuss non-traditional fine grain vertex synchronization without the need to modify Giraph API.…”