Fast Stochastic Block Partitioning via Sampling

Wanye, Frank; Gleyzer, Vitaliy; Feng, Wu-chun

doi:10.1109/hpec.2019.8916542

Cited by 10 publications

(29 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Although this algorithm is slower and harder to parallelize than several alternatives, we focus on it because of its robustness to (a) high variation in community sizes and (b) complex community structure as characterized by high intercommunity connectivity in real-world graphs. This is corroborated by comparing the results obtained by SBP [9,49], Fast-Tracking Resistance [9], Louvain [9,11], and Label Propagation [25] on the Graph Challenge [17] datasets. Across all those implementations, SBP delivered the best and most consistent performance on graphs with complex and heterogeneous structure.…”

Section: Stochastic Block Partitioningsupporting

confidence: 60%

“…Our proposed data-sampling approach for accelerating SBP and, in turn, community detection in general, has been shown to work well on synthetically generated graphs [49]. However, these prior results show performance variations on different graph inputs, variations that have not yet been explainable.…”

Section: Introductionmentioning

confidence: 86%

“…Across all those implementations, SBP delivered the best and most consistent performance on graphs with complex and heterogeneous structure. Furthermore, running the graph-tool SBP implementation [36] on the same graphs led to even better results than those reported in [9,49].…”

Section: Stochastic Block Partitioningmentioning

confidence: 95%

“…The ability of uniform random edge sampling to preserve community structure in graphs was tested in [7], but the community memberships of vertices not present in the sample were not inferred. Our preliminary work [49] also compares several traditional sampling algorithms for the purpose of speeding up the SBP algorithm. The results of these experiments suggest that, given a large enough sample size, even simple random sampling algorithms can preserve community structure.…”

Section: Data Reduction In Graphsmentioning

confidence: 99%

“…The results of these experiments suggest that, given a large enough sample size, even simple random sampling algorithms can preserve community structure. However, the quality of the final community detection results in [49] varied greatly across the different graphs, sample sizes, and sampling algorithms tested, suggesting that some latent factors greatly influence the ability of sampling algorithms to preserve community structure.…”

Section: Data Reduction In Graphsmentioning

confidence: 99%

See 4 more Smart Citations

Topology-Guided Sampling for Fast and Accurate Community Detection

Wanye¹,

Gleyzer²,

Kao³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Community detection is a well-studied problem with applications in domains ranging from computer networking to bioinformatics. While there are many algorithms that perform community detection, the more accurate and statistically robust algorithms tend to be slow and hard to parallelize. One way to speed up such algorithms is through data reduction. However, this approach has not been thoroughly studied, and the quality of results obtained with this approach varies with the graph it is applied to. In this manuscript, we present an approach based on topology-guided sampling for accelerating stochastic block partitioninga community detection algorithm that works well on graphs with complex and heterogeneous community structure. We also introduce a degree-based thresholding scheme that improves the efficacy of our approach at the expense of speedup. Finally, we perform a series of experiments on synthetically generated graphs to determine how various graph parameters affect the quality of results and speedup obtained with our approach, and we validate our approach on real-world data. Our results show that our approach can lead to a speedup of up to 15× over stochastic block partitioning without sampling while maintaining result quality and can even lead to improvements of over 150% in result quality in terms of F1 score on certain kinds of graphs. CCS Concepts: • Information systems → Clustering; • Mathematics of computing → Graph algorithms.

show abstract

Section: Stochastic Block Partitioningsupporting

confidence: 60%

Section: Introductionmentioning

confidence: 86%

Section: Stochastic Block Partitioningmentioning

confidence: 95%

Section: Data Reduction In Graphsmentioning

confidence: 99%

Section: Data Reduction In Graphsmentioning

confidence: 99%

See 3 more Smart Citations

Topology-Guided Sampling for Fast and Accurate Community Detection

Wanye¹,

Gleyzer²,

Kao³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

Exact Distributed Stochastic Block Partitioning

Wanye,

Gleyzer,

Kao

et al. 2023

2023 IEEE International Conference on Cluster Computing (CLUSTER)

Self Cite

View full text Add to dashboard Cite

Stochastic block partitioning (SBP) is a community detection algorithm that is highly accurate even on graphs with a complex community structure, but its inherently serial nature hinders its widespread adoption by the wider scientific community. To make it practical to analyze large real-world graphs with SBP, there is a growing need to parallelize and distribute the algorithm. The current state-of-the-art distributed SBP algorithm is a divide-and-conquer approach that limits communication between compute nodes until the end of inference. This leads to the breaking of computational dependencies, which causes convergence issues as the number of compute nodes increases and when the graph is sufficiently sparse. To address this shortcoming, we introduce EDiSt -an exact distributed stochastic block partitioning algorithm. Under EDiSt, compute nodes periodically share community assignments during inference. Due to this additional communication, EDiSt improves upon the divide-and-conquer algorithm by allowing it to scale out to a larger number of compute nodes without suffering from convergence issues, even on sparse graphs. We show that EDiSt provides speedups of up to 26.9× over the divide-and-conquer approach and speedups up to 44.0× over shared memory parallel SBP when scaled out to 64 compute nodes.

show abstract

On the Multi-Dimensional Acceleration of Stochastic Blockmodeling for Community Detection

Wanye,

Feng

2023

2023 IEEE International Conference on Cluster Computing Workshops (CLUSTER Workshops)

Self Cite

View full text Add to dashboard Cite

Stochastic block partitioning (SBP) is a community detection algorithm that is highly accurate even on graphs with a complex community structure. However, SBP is much slower than more commonly used algorithms, such as Louvain, making SBP impractical for analyzing large real-world graphs with millions of edges. Thus, we aim to realize fast and accurate community detection on large graphs by accelerating the highly accurate SBP algorithm via sampling, parallel and distributed computing on a cluster as well as algorithmic optimization. We compare our approach to other community detection algorithms, showing that SBP accelerated with our methods on 64 compute nodes is up to 1,163× faster than the official "Graph Challenge" baseline SBP implementation, while still being more accurate than the Louvain and Leiden algorithms on large graphs.

show abstract

Fast Stochastic Block Partitioning via Sampling

Cited by 10 publications

References 16 publications

Topology-Guided Sampling for Fast and Accurate Community Detection

Topology-Guided Sampling for Fast and Accurate Community Detection

Exact Distributed Stochastic Block Partitioning

On the Multi-Dimensional Acceleration of Stochastic Blockmodeling for Community Detection

Contact Info

Product

Resources

About