From "think like a vertex" to "think like a graph"

Abstract

To meet the challenge of processing rapidly growing graph and network data created by modern applications, a number of distributed graph processing systems have emerged, such as Pregel and GraphLab. All these systems divide input graphs into partitions, and employ a "think like a vertex" programming model to support iterative graph computation. This vertex-centric model is easy to program and has been proved useful for many graph algorithms. However, this model hides the partitioning information from the users, thus prevents many algorithm-specific optimizations. This often results in longer execution time due to excessive network messages (e.g. in Pregel) or heavy scheduling overhead to ensure data consistency (e.g. in GraphLab). To address this limitation, we propose a new "think like a graph" programming paradigm. Under this graph-centric model, the partition structure is opened up to the users, and can be utilized so that communication within a partition can bypass the heavy message passing or scheduling machinery. We implemented this model in a new system, called Giraph++, based on Apache Giraph, an open source implementation of Pregel. We explore the applicability of the graph-centric model to three categories of graph algorithms, and demonstrate its flexibility and superior performance, especially on well-partitioned data. For example, on a web graph with 118 million vertices and 855 million edges, the graph-centric version of connected component detection algorithm runs 63X faster and uses 204X fewer network messages than its vertex-centric counterpart.

An experimental comparison of pregel-like graph processing systems

“…We consider four categories of graph algorithms: random walk, sequential traversal, parallel traversal, and graph mutation (Table 2) [35].…”

Big Graph Analytics Platforms

“…The vertex-centric framework can be further extended with a blockcentric model (e.g., Giraph++ [Tian et al, 2013] and Blogel [Yan et al, 2014a]), which partitions the vertices into multiple disjoint subgraphs, so that value propagation within each subgraph could bypass network communication. The block-centric model often improves the performance of graph computation by orders of magnitude.…”

“…In addition to faster value propagation (i.e., block-wise), the block-centric model also significantly reduces the communication workload. Representative block-centric system include Giraph++ [Tian et al, 2013] and Blogel [Yan et al, 2014a].…”

“…The vertex-centric programming model of Pregel is also very expressive since a vertex can communicate with any other vertex by passing messages. Since the introduction of Pregel, it has sparked a large number of research works on extending the basic Pregel framework in different aspects to improve the graph processing performance [Tian et al, 2013, Yan et al, 2014a, Han and Daudjee, 2015, Yan et al, 2016b.…”

From Think Parallel to Think Sequential

“…With these comes the need for parallel graph computations. In response to the need, several parallel graph systems have been developed, e.g., Pregel [25], GraphLab [16,24], Trinity [29], GRACE [35], Blogel [37], Giraph++ [31], and GraphX [17].…”