An Elasticity Study of Distributed Graph Processing

Abstract

Graphs are a natural fit for modeling concepts used in solving diverse problems in science, commerce, engineering, and governance. Responding to the variety of graph data and algorithms, many parallel and distributed graph-processing systems exist. However, until now these platforms use a static model of deployment: they only run on a pre-defined set of machines. This raises many conceptual and pragmatic issues, including misfit with the highly dynamic nature of graph processing, and could lead to resource waste and high operational costs. In contrast, in this work we explore a dynamic, elastic model of deployment. To conduct an in-depth elasticity study of distributed graph processing, we build a prototype, JoyGraph, which is the first such system that implements complex, policy-based, and finegrained elasticity. Using the state-of-the-art LDBC Graphalytics benchmark and the SPEC Cloud Group's elasticity metrics, we show the benefits of elasticity in graph processing: improved resource utilization, and aligned operation-workload dynamicity. Furthermore, we explore the cost of elasticity in graph processing. We identify a key drawback: although elasticity does not degrade application throughput, graph-processing workloads are sensitive to data movement while leasing or releasing resources.

Alternative Paradigms of Computation

“…Graph processing systems use massive parallelism or large scale resources of distributed systems in order to process graphs [75]. However, graph applications are not helpful when it comes to static infrastructures, as they often have iterative and highly irregular workloads.…”

Grade10: A Framework for Performance Characterization of Distributed Graph Processing

“…For example, in a top-down graph traversal algorithm, per iteration work is proportional to the size of the traversal-frontier as well as the number of outgoing edges in the frontier [16]. Furthermore, many graph algorithms, like PageRank, dynamically iterate until the output of the algorithm converges, so the number of steps in the algorithm typically depends on the graph structure and per vertex values [17]. Hence, due to the irregular and dynamic nature, modeling graph workload remains challenging.…”

Elasticity in Graph Analytics? A Benchmarking Framework for Elastic Graph Processing

“…How to uncover the costs and benefits of elasticity in distributed graph processing systems? To answer these research questions, we extend our poster [25] by making the following contributions:…”