GAP: Genetic Algorithm Based Large-Scale Graph Partition in Heterogeneous Cluster

Abstract

Graph is an important model to describe various networks, and its scale becomes larger and larger with the development of communication and information technology. The analysis of large-scale graphs requires distributed graph processing systems, and graph partition is the basis of these systems. The existing graph partitioning algorithms are almost proposed for homogeneous clusters, which don't consider the differences among computing nodes in heterogeneous clusters. This paper proposes GAP, a Genetic Algorithm based graph Partitioning algorithm to solve this problem. GAP aims to reduce the total processing time on a heterogeneous cluster by partitioning graphs according to the computing powers of computing nodes. GAP balanced partition the graph initially, and then utilizes genetic algorithm to transfer vertices to reduce cut edges. GAP can balance the processing time of computing nodes, and reduce the communication time among computing nodes. The experiments performed on a heterogeneous cluster demonstrate the outperformance of GAP than Hash.

Dynamic Graph Partitioning Scheme for Supporting Load Balancing in Distributed Graph Environments

“…As graph data have become larger, schemes for dividing and storing large amounts of graph data have been studied. [5][6][7][8]13,[16][17][18][20][21][22][23][24][25][26][27][28][29][30][31]. Research on early graph partitioning schemes has mainly been done on static graphs where data does not change in real time.…”

“…[13]- [16]. If the throughput of a particular node contained in a cluster is high or the memory space is insufficient, the overall system processing performance can decline, and a partitioning strategy is required to store the subgraphs by considering the load state of the nodes [17]- [20]. Therefore, in order to partition dynamic graph data, the load status information of the entire cluster should be managed and determined based on the partitioning policy that considers continuously modified and updated graphs.…”

Graph Computing Systems and Partitioning Techniques: A Survey

“…Researchers have proposed many graph partitioning algorithms in the last decade. The methods of these graph partitioning can be categorized into three: vertex partitioning [41]- [44], edge partitioning [23], [45]- [50], and hybrid partitioning [22], [51]- [53]. These methods can further be classified as offline (in-memory), online (stream), offStream, and dynamic approaches.…”