Adaptive Partitioning for Large-Scale Dynamic Graphs

Abstract

Mining large graphs is critical for several real-world systems such as social networks. Such graphs typically consist of hundreds of millions of vertices and edges and are highly dynamic, with their structure continuously evolving over time. The current data deluge is making this problem even harder: bigger graphs must be processed even faster than before. Appropriate graph partitioning is crucial to scale to large graphs and reduce processing time. However, partitioning large dynamic graphs is challenging: current techniques either do not scale well as they require global graph knowledge or do not handle changes in the graph.In this work, we introduce a general purpose graph partitioning algorithm that adapts to dynamic structural changes in massive graphs. The algorithm works by iteratively adapting the partitions using only local information. We show how the application of our algorithm to three real-world scenarios can speed up the execution of graph analyses by over 50%, while adapting the partitioning to a high volume of changes to the graph.

TAPER: query-aware, partition-enhancement for large, heterogenous graphs

“…The work of Vaquero et al [29] is perhaps the most relevant to our own with TAPER. They propose a system of iterative vertex swapping to adapt to graph changes over time (e.g.…”

“…However, this process is communication and computation intensive and as such is often only used as a "one-off" step, rather than for repeated repartitioning of a graph [29]. Additionally, like other existing partitioners, ParMetis is agnostic to a changing query workload.…”

“…Yet more partitioning systems [29,31] consider other aspects of a graph's workload, besides edge traversals. Xu et al [31] note that, for many BSP workloads, reducing the total communication time of a workload by simply minimising a graph's inter-partition edges may not be an appropriate optimisation.…”

“…Not only are all these systems [3,22,25,[29][30][31] focused on different types of workload than we are with TAPER, but in an online setting, workload summaries which directly measure individual edge traversals [3,25,30] are vulnerable to overfitting. Path queries in particular are often only executed over a subset of the graph, i.e.…”

GraSP: distributed streaming graph partitioning

“…For networks with dynamic structure, iterative approaches can dynamically adjust the partitions to suit changing graph structure. Vaquero et al propose a method for iteratively adjusting graph partitions to cope with changes in the graph, using only local information [24]. This work demonstrated the power and scalability of leveraging local data to improve partition quality, especially to reduce the edges cut.…”