Ripple Walk Training: A Subgraph-based training framework for Large and Deep Graph Neural Network

Abstract

Graph neural networks (GNNs) have achieved outstanding performance in learning graph-structured data. Many current GNNs suffer from three problems when facing large-size graphs and using a deeper structure: neighbors explosion, node dependence, and oversmoothing. In this paper, we propose a general subgraph-based training framework, namely Ripple Walk Training (RWT), for deep and large graph neural networks. RWT samples subgraphs from the full graph to constitute a mini-batch and the full GNN is updated based on the mini-batch gradient. We analyze the high-quality subgraphs required in a mini-batch in a theoretical way. A novel sampling method Ripple Walk Sampler works for sampling these high-quality subgraphs to constitute the mini-batch, which considers both the randomness and connectivity of the graph-structured data. Extensive experiments on different sizes of graphs demonstrate the effectiveness of RWT in training various GNNs (GCN & GAT).

Sampling Methods for Efficient Training of Graph Convolutional Networks: A Survey

“…Sampling methods used in works [26,[55][56] all satisfy the mechanism that subgraphs are induced from nodes (edges) extension. methods have many differences in various aspects.…”

“…RWT [55], Parallelized Graph Sampling [56] Heterogeneous sampling method Time-related sampling [61], HetGNN [62],…”

“…For sampling methods proposed in the works [17,[19][20][21], the sampling operation is applied to each node's neighbors. A part of neighbors of a single node are sampled in one sampling batch, so we define this kind of sampling method as the node-wise sampling method; for sampling methods proposed in the works [22][23][24], multiple nodes are sampled simultaneously in the sampling operation in each layer, so we define this kind of sampling method as the layer-wise sampling method; for sampling methods proposed in the works [25,26,[55][56], a subgraph that induced from specially chosen nodes (and edges) is sampled in one sampling batch for further computation, so we define this kind of sampling method as the subgraph-based sampling method; for sampling methods proposed in the works [61][62][63][64], different types of nodes and edges are sampled in a heterogeneous graph. This kind of sampling method generally varies with different structures of heterogeneous graphs, and we therefore define this kind of sampling method as the heterogeneous sampling method.…”

Scalable Graph Neural Network Training: The Case for Sampling

“…Subgraph-based sampling samples a subgraph, which is composed of selected vertices and edges, and conducts training using the subgraph. Existing work generate subgraphs by either partitioning the whole graph or extending vertices and edges using specific policies [2,8,39]. For example, Clus-terGCN [8] first partitions the whole graph into multiple clusters using graph clustering algorithms and then randomly samples a fixed number of clusters as a mini-batch by combining these clusters into a subgraph.…”

Two-level Graph Neural Network

“…For example, specific patterns of atoms or modes of interaction can be discovered by identifying specific subgraph topologies. Bai et al [12] propose a general subgraph-based training framework referred to as Ripple Walk Training (RWT). This can not only accelerate the training speed on large graphs but also solve problems associated with the memory bottleneck.…”