Elasticity in Graph Analytics? A Benchmarking Framework for Elastic Graph Processing

Abstract

Graphs are a natural fit for modeling concepts used in solving diverse problems in science, commerce, engineering, and governance. Responding to the diversity of graph data and algorithms, many parallel and distributed graph-processing systems exist. However, until now these platforms use a static model of deployment: they only run on a pre-defined set of machines. This raises many conceptual and pragmatic issues, including misfit with the highly dynamic nature of graph processing, and could lead to resource waste and high operational costs. In contrast, in this work we explore the benefits and drawbacks of the dynamic model of deployment. Building a threelayer benchmarking framework for assessing elasticity in graph analytics, we conduct an in-depth elasticity study of distributed graph processing. Our framework is composed of state-ofthe-art workloads, autoscalers, and metrics, derived from the LDBC Graphalytics benchmark and SPEC RG Cloud Group's elasticity metrics. We uncover the benefits and cost of elasticity in graph processing: while elasticity allows for fine-grained resource management, and does not degrade application performance, we find that graph workloads are sensitive to data migration while leasing or releasing resources. Moreover, we identify non-trivial interactions between scaling policies and graph workloads, which add an extra level of complexity to resource management and scheduling for graph processing.

Time-Efficient and High-Quality Graph Partitioning for Graph Dynamic Scaling

“…The dynamic scaling for the traditional vertex graph partitioning has been studied in some work [16][17][18][19]. The main difference from these efforts is that our proposal is based on edge partitioning.…”

“…In the case of distributed graph analysis, however, scaling the number of graph partitions efficiently while achieving high quality is a challenging endeavor due to the trade-off between efficiency and quality. On the one hand, several dynamic scaling methods based on efficient graph partitioning have been proposed [16][17][18][19][20], which, however, exhibit limited quality. It results in high communication costs, affecting the performance of distributed graph processing, as shown in the top of Figure 1.…”

“…In this paper, we propose a novel approach to the dynamic scaling of graph partitions, which enables us to efficiently recompute the partitioning when the number of partitions is changed while keeping partitioning quality high. As with the latest work [20], we focus on (vertex-cut) edge partitioning rather than traditional (edge-cut) vertex partitioning, as discussed in the other existing work [16][17][18][19]. The edge partitioning is known to provide a better workload balance because the computational cost in the graph processing essentially depends on the number of edges rather than that of vertices [2,3].…”

Graphless: Toward Serverless Graph Processing

“…R2 Fine-grained elasticity. To process efficiently the highly irregular workloads of graph processing, the system must be elastic [18], that is, be able to scale-up and -down seamlessly with the graph workload. Moreover, to avoid excessive overprovisioning, elasticity should be fine-grained.…”

“…Second, distributed graph processing systems generally run on a fixed number of compute nodes, making such systems brittle and unable to respond to changes in the workload by scaling-in or -out; the systems are either over-or under-provisioned. In practice, under-provisioning leads either to crashing or thrashing, and over-provisioning leads to resource waste due to the irregularity of graph processing workloads [18].…”