Communication-Free Massively Distributed Graph Generation

Abstract

Analyzing massive complex networks yields promising insights about our everyday lives. Building scalable algorithms to do so is a challenging task that requires a careful analysis and an extensive evaluation. However, engineering such algorithms is often hindered by the scarcity of publicly available datasets.Network generators serve as a tool to alleviate this problem by providing synthetic instances with controllable parameters. However, many network generators fail to provide instances on a massive scale due to their sequential nature or resource constraints. Additionally, truly scalable network generators are few and often limited in their realism.In this work, we present novel generators for a variety of network models that are frequently used as benchmarks. By making use of pseudorandomization and divide-and-conquer schemes, our generators follow a communication-free paradigm. The resulting generators are thus embarrassingly parallel and have a near optimal scaling behavior. This allows us to generate instances of up to 2 43 vertices and 2 47 edges in less than 22 minutes on 32 768 cores. Therefore, our generators allow new graph families to be used on an unprecedented scale.

Linear work generation of R-MAT graphs

“…The speed of our generators compares favorably to generators for other models. For example, the sequential Erdős-Renyi generator of Funke et al (2019) is only around 10% faster than our sequential generator, but for a much simpler model (the special case a = b = c = d). On the other hand, the streaming hyperbolic graph generator sRHG of Funke et al (2019), a more complicated model, takes around 5 times more time per edge.…”

“…However, recent results on communication-free graph generation (Sanders & Schulz, 2016;Funke et al, 2019;Bläsius et al, 2019) put the role of R-MAT as the most scalable graph generator into question. Other models with similar properties, such as Barabasi-Albert (BA) preferential attachment graphs (Barabasi & Albert, 1999;Sanders & Schulz, 2016) or random hyperbolic graphs (RHGs) (Krioukov et al, 2010;Funke et al, 2019;Bläsius et al, 2019) can now be generated in a massively parallel fashion using only linear work, whereas previous R-MAT generators need logarithmic time for each edge. Indeed, the highly tuned RHG implementation in Funke et al (2019) outperforms previous R-MAT generators also in practice.…”

“…Other models with similar properties, such as Barabasi-Albert (BA) preferential attachment graphs (Barabasi & Albert, 1999;Sanders & Schulz, 2016) or random hyperbolic graphs (RHGs) (Krioukov et al, 2010;Funke et al, 2019;Bläsius et al, 2019) can now be generated in a massively parallel fashion using only linear work, whereas previous R-MAT generators need logarithmic time for each edge. Indeed, the highly tuned RHG implementation in Funke et al (2019) outperforms previous R-MAT generators also in practice. This is surprising since the R-MAT model is very simple while generating RHGs requires comparatively sophisticated geometric data structures and evaluating transcendental functions.…”

Novel Parallel Algorithms for Fast Multi-GPU-Based Generation of Massive Scale-Free Networks

“…Due to its simpler parallel implementation, the Graph500 group [1] choose the SKG model in their supercomputer benchmark. Highly scalable generators for Erdős-Renyi, 2D/3D random geometric graphs, 2D/3D Delaunay graphs, and hyperbolic random graphs are described in [15]. The corresponding software library release also includes an implementation of the algorithm described in [24].…”

Balanced k-means for Parallel Geometric Partitioning

“…We also generated ve 3D Delaunay triangulations from approx. 1M to 16M vertices using the generator of Funke et al [19]. e graphs alyaTestCaseA with 9.9 million vertices and alyaTestCaseB with 30.9 million vertices (representing the respiratory system) are from the PRACE benchmark suite [37].…”