MapReduce-Based Pattern Finding Algorithm Applied in Motif Detection for Prescription Compatibility Network

Liu, Yang; Jiang, Xiaohong; Chen, Huajun; Ma, Jun; Zhang, Xiangyu

doi:10.1007/978-3-642-03644-6_27

Cited by 31 publications

(28 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…MapReduce/Hadoop has become a popular approach for parallel computing, and graph algorithms (e.g., [27], [21], [26], [19] for subgraph enumeration) are being developed using this approach. Among these, [27], [21], [26] develop algorithms for enumerating triangles and give worst case O(|E|) work complexity bounds for this problem, using Hadoop.…”

Section: Related Workmentioning

confidence: 99%

“…It supports subgraphs in the form of any labeled tree. As discussed earlier, the only prior Hadoop based approaches have been on triangles [27], [21], [26] in very large networks, or more general subgraphs on relatively small networks [19]. Our main technical contribution is the development of a Hadoop version of the color coding algorithm of Alon et al [1], [2], which is a (sequential) randomized approximation algorithm for subgraph counting.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

SAHAD: Subgraph Analysis in Massive Networks Using Hadoop

Zhao

Wang

Butt

et al. 2012

2012 IEEE 26th International Parallel and Distributed Processing Symposium

View full text Add to dashboard Cite

Abstract-Relational subgraph analysis, e.g. finding labeled subgraphs in a network, which are isomorphic to a template, is a key problem in many graph related applications. It is computationally challenging for large networks and complex templates. In this paper, we develop SAHAD, an algorithm for relational subgraph analysis using Hadoop, in which the subgraph is in the form of a tree. SAHAD is able to solve a variety of problems closely related with subgraph isomorphism, including counting labeled/unlabeled subgraphs, finding supervised motifs, and computing graphlet frequency distribution. We prove that the worst case work complexity for SAHAD is asymptotically very close to that of the best sequential algorithm. On a mid-size cluster with about 40 compute nodes, SAHAD scales to networks with up to 9 million nodes and a quarter billion edges, and templates with up to 12 nodes. To the best of our knowledge, SAHAD is the first such Hadoop based subgraph/subtree analysis algorithm, and performs significantly better than prior approaches for very large graphs and templates. Another unique aspect is that SAHAD is also amenable to running quite easily on Amazon EC2, without needs for any system level optimization.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

SAHAD: Subgraph Analysis in Massive Networks Using Hadoop

Zhao

Wang

Butt

et al. 2012

2012 IEEE 26th International Parallel and Distributed Processing Symposium

View full text Add to dashboard Cite

show abstract

“…Hadoop MapReduce [12] is open-source software known for its fault tolerance, scalability and ease of use. Liu et al [13] has introduced Map Reduce-Based Pattern Finding Algorithm (MRPF) which is a parallel solution for finding frequent subgraph patterns to improve upon the performance. But the statistical significance testing has not been implemented in this algorithm.…”

Section: Fig 2 (A) Exponential Increase In Subgraphs When Size Of Nmentioning

confidence: 99%

Network Motif Analysis in Clouds - Subgraph Enumeration with Iterative Hadoop MapReduce

2016

STCC

View full text Add to dashboard Cite

Finding network motifs in biological networks is a computationally intensive task as it involves traversing through a large network to enumerate all possible subgraphs of a given size, and then determining their statistical uniqueness by sampling subgraphs from a large number (more than 1,000) of random graph pools. There have been parallelization efforts in the past to mitigate the computational intensity for finding network motifs. However, they are either more for the frequent subgraphs in networks rather than network motifs, or require complex manipulations on message passing or designs. Additionally, most of the parallel algorithms are unavailable as tools for use. Here, we introduce a project of 'complete network motif parallelization.' This project aims to improve performance of serial algorithms for finding all subgraph patterns with given size in PPI networks by parallelizing them using iterative Hadoop MapReduce on Google Cloud, and then determining their uniqueness through explicit or direct random graph generation to determine network motifs. In this paper, we describe the parallelization of subgraph enumeration and McKay's canonical algorithms and present the experimental results with a significant improvement in performance, up to 37 times speedup. We are continuing to parallelize network motif significance as a next step, and expecting the completion of the project in near future.

show abstract

“…Nevertheless, most of them are sequential algorithms that require much time to mine large datasets, including SiGraM (Single Graph Mining) [10], GERM (Graph Evolution Rule Miner) [11] and GRAMI (Graph Mining) [12]. Meanwhile, researchers have also used parallel and distributed computing techniques to accelerate the computation, in which two parallel computing frameworks are mainly used: Map-Reduce [13][14][15][16][17][18] and MPI (Message Passing Interface) [19]. The existing MapReduce implementations of parallel FSM algorithms are all based on Hadoop [20] and are designed for graph transaction and not for a single graph, often reaching IO (Input and Output) bottlenecks because they have to spend a lot of time moving the data/processes in and out of the disk during iteration of the algorithms.…”

Section: Introductionmentioning

confidence: 99%

“…The existing MapReduce implementations of parallel FSM algorithms are all based on Hadoop [20] and are designed for graph transaction and not for a single graph, often reaching IO (Input and Output) bottlenecks because they have to spend a lot of time moving the data/processes in and out of the disk during iteration of the algorithms. Besides, some of these algorithms cannot support mining via subgraph extension [14,15]. That is to say, users must provide the size of subgraph as input.…”

Section: Introductionmentioning

confidence: 99%

A Parallel Approach for Frequent Subgraph Mining in a Single Large Graph Using Spark

Qiao

Zhang

et al. 2018

Applied Sciences

View full text Add to dashboard Cite

Frequent subgraph mining (FSM) plays an important role in graph mining, attracting a great deal of attention in many areas, such as bioinformatics, web data mining and social networks. In this paper, we propose SSIGRAM (Spark based Single Graph Mining), a Spark based parallel frequent subgraph mining algorithm in a single large graph. Aiming to approach the two computational challenges of FSM, we conduct the subgraph extension and support evaluation parallel across all the distributed cluster worker nodes. In addition, we also employ a heuristic search strategy and three novel optimizations: load balancing, pre-search pruning and top-down pruning in the support evaluation process, which significantly improve the performance. Extensive experiments with four different real-world datasets demonstrate that the proposed algorithm outperforms the existing GRAMI (Graph Mining) algorithm by an order of magnitude for all datasets and can work with a lower support threshold.

show abstract

MapReduce-Based Pattern Finding Algorithm Applied in Motif Detection for Prescription Compatibility Network

Cited by 31 publications

References 20 publications

SAHAD: Subgraph Analysis in Massive Networks Using Hadoop

SAHAD: Subgraph Analysis in Massive Networks Using Hadoop

Network Motif Analysis in Clouds - Subgraph Enumeration with Iterative Hadoop MapReduce

A Parallel Approach for Frequent Subgraph Mining in a Single Large Graph Using Spark

Contact Info

Product

Resources

About