Parallel community detection on large graphs with MapReduce and GraphChi

Moon, Seunghyeon; Lee, Jae-Gil; Kang, Minseo; Choy, Minsoo; Lee, JinWoo

doi:10.1016/j.datak.2015.05.001

Cited by 30 publications

(10 citation statements)

References 17 publications

(12 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For instance, the work from Lu, Halappanavar and Kalyanaraman [58] describes new heuristics which do not required sequential computations, and thus enable the parallelisation of Louvain method achieving speedups of up to 169 using 32 threads. Moon et al [63] also perform parallel community detection, in this case by proposing two new algorithms, namely SPB-MRA and SPB-VCA to circumvent the main pitfalls of Girvan-Newman divisive algorithm, the first one using the MapReduce distributed framework and the second one using GraphChi, a largescale graph computation system. Finally, Clementi et al [21] propose a distributed protocol based on label-propagation for performing community detection in dynamic graphs.…”

Section: Community Detectionmentioning

confidence: 97%

Beyond social graphs: mining patterns underlying social interactions

Baldominos

Calle

Cuadra

2016

Pattern Anal Applic

View full text Add to dashboard Cite

This work aims at discovering and extracting relevant patterns underlying social interactions. To do so, some knowledge extracted from Facebook, a social networking site, is formalised by means of an Extended Social Graph, a data structure which goes beyond the original concept of a social graph by also incorporating information on interests. When the Extended Social Graph is built, state-of-the-art techniques are applied over it in order to discover communities. Once these social communities are found, statistical techniques will look for relevant patterns common to each of those, in such a way that each cluster of users is characterised by a set of common features. The resulting knowledge will be used to develop and evaluate a social recommender system, which aims at suggesting users in a social network with possible friends or interests.

show abstract

Section: Community Detectionmentioning

confidence: 97%

Beyond social graphs: mining patterns underlying social interactions

Baldominos

Calle

Cuadra

2016

Pattern Anal Applic

View full text Add to dashboard Cite

show abstract

“…IncDiSC 24 formulates the single-linkage hierarchical clustering problem as a Minimum Spanning Tree (MST) construction problem on a complete graph and implement the uni¯ed algorithm by employing MapReduce framework. Shortest Path Betweenness MapReduce Algorithm (SPB-MRA) 25 is a parallel version of a divisive hierarchical clustering algorithm for community detection based on the MapReduce model. Sun et al, 26 presents an e±cient hierarchical clustering method of mining large datasets with MapReduce which includes two optimization techniques: Batch updating to reduce the computational time and communication cost among cluster nodes, and co-occurrence-based feature selection to decrease the dimension of feature vectors and eliminate noise features.…”

Section: Clustering For Large Datasetsmentioning

confidence: 99%

DHC: A Distributed Hierarchical Clustering Algorithm for Large Datasets

Zhang

Chen

et al. 2019

J CIRCUIT SYST COMP

View full text Add to dashboard Cite

Hierarchical clustering is a classical method to provide a hierarchical representation for the purpose of data analysis. However, in practical applications, it is difficult to deal with massive datasets due to their high computation complexity. To overcome this challenge, this paper presents a novel distributed storage and computation hierarchical clustering algorithm, which has a lower time complexity than the standard hierarchical clustering algorithms. Our proposed approach is suitable for hierarchical clustering on massive datasets, which has the following advantages. First, the algorithm is able to store massive dataset exceeding the main memory space by using distributed storage nodes. Second, the algorithm is able to efficiently process nearest neighbor searching along parallel lines by using distributed computation at each node. Extensive experiments are carried out to validate the effectiveness of the DHC algorithm. Experimental results demonstrate that the algorithm is 10 times faster than the standard hierarchical clustering algorithm, which is an effective and flexible distributed algorithm of hierarchical clustering for massive datasets.

show abstract

“…In another work, Zhang and Xiong [8] followed the same approach for the search of dynamic path in large road network based on cloud computing. In addition, Seunghyeon et al [27] proposed a parallel version of Girvan-Newman algorithm based on the concept of Hadoop MapReduce to improve the computational time in large-scale network.…”

Section: Distributed Shortest Paths Computingmentioning

confidence: 99%

The MapReduce-based approach to improve the shortest path computation in large-scale road networks: the case of A* algorithm

et al. 2018

View full text Add to dashboard Cite

With the increasing size of road networks (4.4 billions of vertices and 6 billions of uploaded GPS points, according to OpenStreetMap data stats 2018 [1]), there has been vast improvement in hardware architecture for intelligent transportation system. The traditional GPS systems embedded in vehicles are only designed to find the shortest paths in small or medium road networks. With the increase in the size of road-networks, implementing efficient GPS programs has become challenging. This is mainly due to impractical computational time taken to compute the optimal path. The basic algorithms used for the Single Source Shortest Path Problem (SSSPP) are not suited for intensive computation in large-scale networks because of long latency time. This is one of the crucial problem of route-guidance systems for highway vehicles including the Vehicle Routing Problem (VPR), Traveling Salesman Problem (TSP) and Pickup and Delivery Problem (PDP). Currently, there are lot of approach of SSSPP such as label setting, dynamic programming, heuristic and/or bidirectional heuristic. However, they are inefficient

show abstract

Parallel community detection on large graphs with MapReduce and GraphChi

Cited by 30 publications

References 17 publications

Beyond social graphs: mining patterns underlying social interactions

Beyond social graphs: mining patterns underlying social interactions

DHC: A Distributed Hierarchical Clustering Algorithm for Large Datasets

The MapReduce-based approach to improve the shortest path computation in large-scale road networks: the case of A* algorithm

Contact Info

Product

Resources

About