Ripple Walk Training: A Subgraph-based Training Framework for Large and Deep Graph Neural Network

Bai, Jiyang; Ren, Yuxiang; Zhang, Jiawei

doi:10.1109/ijcnn52387.2021.9533429

Cited by 16 publications

(43 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In such a case, the depth of GNN models can reach more than fifty with a better performance. Till now, the optimized methods based on subgraph learning are limited [14,6,4]. Their measurement of the quality of sampled subgraphs is not comprehensive enough, and the sampling of subgraphs is precisely the key to this type of method.…”

Section: Oversmoothingmentioning

confidence: 99%

“…The learning scheme shown in Equation 1 needs to take the full graph G as input. Training GNN models with the full graph, especially facing large-sized graphs, may easily lead to aforementioned three problems:neighbors explosion, node dependence, and oversmoothing [4]. To solve these problems, subgraph-based training methods [4,14] employ subgraphs of G to construct the mini-batch in each training iteration and update the complete GNN models based on the mini-batch gradient.…”

Section: Subgraph-based Training For Gnn Modelsmentioning

confidence: 99%

“…GNNs learn high-level representations through a recursive neighbors aggregation scheme [5], which causes the number of neighbors to explode. This problem is described as neighbors explosion [4], which leads to exponentially-growing computation complexity. Because neighboring nodes are interdependent in the learning process, most current GNNs have to work based on the full graph.…”

Section: Introductionmentioning

confidence: 99%

“…The other direction is the graph-sampling method [6,14,4], which can deal with the memory overhead of loading the full graph. Cluster-GCN [6] constructs the subgraphs mini-batch by clustering on the full graph to train the GCN.…”

Section: Introductionmentioning

confidence: 99%

“…This metric we define in this paper is Connection Failure Distance, which is utilized by MeGuide Sampler to control the longest multi-hop connection in subgraphs. In this way, MeGuide can help GNNs avoid unexpected aggregations (e.g., aggregating nodes with different labels), which is the primary cause of oversmoothing [4]. To a certain extent, the size of subgraphs can also be determined, but in other sampling methods, it is a hyper-parameter.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Measuring and Sampling: A Metric-guided Subgraph Learning Framework for Graph Neural Network

Bai¹,

Ren²,

Zhang³

2021

Preprint

Self Cite

View full text Add to dashboard Cite

Section: Oversmoothingmentioning

confidence: 99%

Section: Subgraph-based Training For Gnn Modelsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Measuring and Sampling: A Metric-guided Subgraph Learning Framework for Graph Neural Network

Bai¹,

Ren²,

Zhang³

2021

Preprint

Self Cite

View full text Add to dashboard Cite

Adaptive momentum with discriminative weight for neural network stochastic optimization

Bai

Ren

Zhang

2022

Int J of Intelligent Sys

Self Cite

View full text Add to dashboard Cite

Optimization algorithms with momentum have been widely used for building deep learning models because of the fast convergence rate. Momentum helps accelerate Stochastic gradient descent in relevant directions in parameter updating, minifying the oscillations of the parameters update route. The gradient of each step in optimization algorithms with momentum is calculated by a part of the training samples, so there exists stochasticity, which may bring errors to parameter updates. In this case, momentum placing the influence of the last step to the current step with a fixed weight is obviously inaccurate, which propagates the error and hinders the correction of the current step. Besides, such a hyperparameter can be extremely hard to tune in applications as well. In this paper, we introduce a novel optimization algorithm, namely, Discriminative wEight on Adaptive Momentum (DEAM). Instead of assigning the momentum term weight with a fixed hyperparameter, DEAM proposes to compute the momentum weight automatically based on the discriminative angle. The momentum term weight will be assigned with an appropriate value that configures momentum in the current step. In this way, DEAM involves fewer hyperparameters. DEAM also contains a novel backtrack term, which restricts redundant updates when the correction of the last step is needed. The backtrack term can effectively adapt the learning rate and achieve the anticipatory update as well. Extensive experiments demonstrate that DEAM can achieve a faster convergence rate than the existing optimization algorithms in training the deep learning models of both convex and nonconvex situations.

show abstract

Measuring and sampling: A metric‐guided subgraph learning framework for graph neural network

Bai

Ren

Zhang

2022

Int J of Intelligent Sys

Self Cite

View full text Add to dashboard Cite

Graph neural networks (GNNs) have shown convincing performance in learning powerful node representations that preserve both node attributes and graph structural information. However, many GNNs encounter problems in effectiveness and efficiency when they are designed with a deeper network structure or handle large-sized graphs. Several sampling algorithms have been proposed for improving and accelerating the training of GNNs, yet they ignore understanding the source of GNNs performance gain. The measurement of information within graph data can help the sampling algorithms to keep high-value information while removing redundant information and even noise. In this paper, we propose a Metric-Guided (MeGuide) subgraph learning framework for GNNs. MeGuide employs two novel metrics: Feature Smoothness and Connection Failure Distance to guide the subgraph sampling and mini-batch based training. Feature Smoothness is designed for analyzing the feature of nodes to retain the most valuable information, while Connection Failure Distance can measure the structural information to control the size of subgraphs. We demonstrate the effectiveness and efficiency of MeGuide in training various GNNs on multiple data sets.

show abstract

Ripple Walk Training: A Subgraph-based Training Framework for Large and Deep Graph Neural Network

Cited by 16 publications

References 11 publications

Measuring and Sampling: A Metric-guided Subgraph Learning Framework for Graph Neural Network

Measuring and Sampling: A Metric-guided Subgraph Learning Framework for Graph Neural Network

Adaptive momentum with discriminative weight for neural network stochastic optimization

Measuring and sampling: A metric‐guided subgraph learning framework for graph neural network

Contact Info

Product

Resources

About