Partitioning big graph with respect to arbitrary proportions in a streaming manner

Hu, Kekun; Zeng, Guosun; Jiang, Huowen; Wang, Wei

doi:10.1016/j.future.2017.06.027

Cited by 9 publications

(7 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…2) The total communication time along edges within this partition. Give a graph problem, its computation and communication workloads can be assessed [26]. Let ( = 1,2, ⋯ , ) be the computation and communication workload of partition , and…”

Section: Problem Statementsmentioning

confidence: 99%

“…Taking the workload measurement proposed in [26] as an example, of the Dijkstra's single source shortest path (SSSP) algorithm is…”

Section: Problem Statementsmentioning

confidence: 99%

“…Because GAP algorithm is based on a specific graph algorithm, we apply SSSP to analyze the performance. According to [26], the parameters in Figures 1 and 2 also show that GAP algorithm can partition the graph according the computing power of computing nodes: the stronger the computing nodes are, the more vertices and edges are allocated. In particular, the number of vertices of each computing node is proportional to their computing power.…”

Section: B Partition Optimization By Genetic Algorithmmentioning

confidence: 99%

See 2 more Smart Citations

GAP: Genetic Algorithm Based Large-Scale Graph Partition in Heterogeneous Cluster

Cui

Zhou

et al. 2020

IEEE Access

View full text Add to dashboard Cite

Graph is an important model to describe various networks, and its scale becomes larger and larger with the development of communication and information technology. The analysis of large-scale graphs requires distributed graph processing systems, and graph partition is the basis of these systems. The existing graph partitioning algorithms are almost proposed for homogeneous clusters, which don't consider the differences among computing nodes in heterogeneous clusters. This paper proposes GAP, a Genetic Algorithm based graph Partitioning algorithm to solve this problem. GAP aims to reduce the total processing time on a heterogeneous cluster by partitioning graphs according to the computing powers of computing nodes. GAP balanced partition the graph initially, and then utilizes genetic algorithm to transfer vertices to reduce cut edges. GAP can balance the processing time of computing nodes, and reduce the communication time among computing nodes. The experiments performed on a heterogeneous cluster demonstrate the outperformance of GAP than Hash.

show abstract

Section: Problem Statementsmentioning

confidence: 99%

“…Taking the workload measurement proposed in [26] as an example, of the Dijkstra's single source shortest path (SSSP) algorithm is…”

Section: Problem Statementsmentioning

confidence: 99%

Section: B Partition Optimization By Genetic Algorithmmentioning

confidence: 99%

See 1 more Smart Citation

GAP: Genetic Algorithm Based Large-Scale Graph Partition in Heterogeneous Cluster

Cui

Zhou

et al. 2020

IEEE Access

View full text Add to dashboard Cite

show abstract

“…This objective is equivalent to balancing the workload across computational nodes of ARC. Considering the heterogeneity of ARC, in the mapping phase we design a mapping algorithm named HALB (Heterogeneous_Aware_Load_ Balancing) with the aim of minimizing the general load balance factor [1]. It is defined as ρ = max…”

Section: Heterogeneous-aware Mapping Algorithmmentioning

confidence: 99%

“…The size of modern graph datasets generated from various domains such as social networks, e-Commerce, the Web, and smart cities, is very big and increasing dramatically [1]. For example, as a famous social network, Facebook had over 2.32 billion vertices and 179 billion edges as of December, 2018 [2].…”

Section: Introductionmentioning

confidence: 99%

Cluster-Scheduling Big Graph Traversal Task for Parallel Processing in Heterogeneous Cloud Based on DAG Transformation

Zeng

Ding

et al. 2019

IEEE Access

View full text Add to dashboard Cite

Task scheduling is the key to the full utilization of heterogeneous cloud capabilities for parallel processing of big graphs. Most graph processing systems adopt single-granularity scheduling mechanisms without considering the heterogeneity of the cloud, leading to poor performance. To alleviate it by learning from the excellent directed acyclic graph (DAG)-based scheduling techniques accumulated in traditional parallel computing, we first present a streaming DAG-construction heuristic. It transforms a big graph along with graph traversal algorithms to be carried out into a DAG. We then propose a three-phase heterogeneousaware cluster-scheduling algorithm to schedule the DAG into a heterogeneous cloud for parallel processing. In the first phase, we design a parallel linear clustering algorithm to cluster the DAG into a series of linear clusters with different granularities. In the second phase, we design a heterogeneous-aware load balancing algorithm to map these clusters to different computational nodes of the cloud. In the last phase, we design a task ordering algorithm to assigns these clusters as-early-as-possible start times. The experimental results show that our scheme can generate high-quality schedules and improve the efficiency and performance of parallel processing of big graphs in the heterogeneous cloud. INDEX TERMS Heterogeneous cloud, big graph traversal task, parallel processing, DAG, clusterscheduling, granularity.

show abstract

DETER: Streaming Graph Partitioning via Combined Degree and Cluster Information

Jiang

2020

Algorithms and Architectures for Parallel Processing

View full text Add to dashboard Cite

Partitioning big graph with respect to arbitrary proportions in a streaming manner

Cited by 9 publications

References 19 publications

GAP: Genetic Algorithm Based Large-Scale Graph Partition in Heterogeneous Cluster

GAP: Genetic Algorithm Based Large-Scale Graph Partition in Heterogeneous Cluster

Cluster-Scheduling Big Graph Traversal Task for Parallel Processing in Heterogeneous Cloud Based on DAG Transformation

DETER: Streaming Graph Partitioning via Combined Degree and Cluster Information

Contact Info

Product

Resources

About