GoFFish: A Sub-graph Centric Framework for Large-Scale Graph Analytics

Abstract

Large scale graph processing is a major research area for Big Data exploration. Vertex centric programming models like Pregel are gaining traction due to their simple abstraction that allows for scalable execution on distributed systems naturally. However, there are limitations to this approach which cause vertex centric algorithms to under-perform due to poor compute to communication overhead ratio and slow convergence of iterative superstep. In this paper we introduce GoFFish a scalable sub-graph centric framework co-designed with a distributed persistent graph storage for large scale graph analytics on commodity clusters. We introduce a sub-graph centric programming abstraction that combines the scalability of a vertex centric approach with the flexibility of shared memory sub-graph computation. We map Connected Components, SSSP and PageRank algorithms to this model to illustrate its flexibility. Further, we empirically analyze GoFFish using several real world graphs and demonstrate its significant performance improvement, orders of magnitude in some cases, compared to Apache Giraph, the leading open source vertex centric implementation.

Empowering Fast Incremental Computation over Large Scale Dynamic Graphs

“…This observation is consistent with slimier observations reported in other studies [8]. In our proof of concept implementation we used static graph partitioning algorithms to show the performance advantages of our approach.…”

“…This model can be thought of as an extension to the subgraph/partition centric models proposed in [8], [9] where local computation within the partition is executed using vertex centric model. Vertices within partitions are executed in parallel using the multiple cores in each worker machine.…”

“…In our paper, we showed that our HBSP model, when augmented with memorization can significantly outperform GraphInc. While memorization is applicable to recently proposed subgraph or partition-centric [8], [9], [10]. However, since these models do not exert control at the vertex level, fine graph computation reuse cannot be achieved.…”

“…Recent work on subgraphcentric programming models have reduced the communication (number of messages) and synchronization (i.e. number of iterations required to converge to a solution) overhead of vertex-centric programming model [8], [9], [10].…”

Real-Time Analytics for Fast Evolving Social Graphs

“…Existing graph processing methods have focused either on large shared memory approaches where the graph is streamed and processed in-memory [6], or on batch processing techniques for distributed computing where periodic graph snapshots are taken and processed independently [9]. While these techniques are efficient for some classes of evolving graphs where data is either centrally stored or their structure is slow changing, present day online social networks where data is gathered in geographically distributed data centers (i.e., on the cloud) and which exhibit fast topology changes.…”