PairNorm: Tackling Oversmoothing in GNNs

Zhao, Lingxiao; Akoglu, Leman

doi:10.48550/arxiv.1909.12223

Cited by 50 publications

(87 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…(2) DropEdge and PairNorms (including PairNorm-SI and -SCS) are recently proposed state-of-the-art methods to relieve over-smoothness, whose performance does not drop as fast as conventional GNNs. In particular, the performance of DropEdge is comparable to its backbone GCN while PairNorms is inferior to its backbone in many scenarios, which is consistent with the observations in the original papers [51].…”

Section: Resultssupporting

confidence: 88%

“…Consequently, it remains a challenging problem to directly relieve over-smoothing in the information propagating process. To the best of our knowledge, we are the first to provide an understanding of this problem from the perspective of the spectral filter, and our experiments also demonstrate its superiority over other prevalent solutions such as DropEdge [38] and PairNorm [51].…”

Section: Resultsmentioning

confidence: 67%

“…Nevertheless, such methods are unable to avoid the situation where a node is overwhelmed in the information of its neighbors in the information aggregation process. There are also works directly relieve over-smoothing in this process via using either edge masking [38] or re-normalization [51]; however performance still obviously reduces when models go deeper. Consequently, it remains a challenging problem to directly relieve over-smoothing in the information propagating process.…”

Section: Resultsmentioning

confidence: 99%

“…These two methods are compared with the following state-of-theart GNNs: (1) GCN [24]; (2) GraphSAGE [19]; (3) SGC [43]. Also, we compare our framework with the recently developed methods that tackle over-smoothing: (4) DropEdge [38] -which relieves oversmoothing issue by edge masking; (5) PairNorm [51] -which tackles over-smoothing with a normalization layer and its two different corresponding implementations are PairNorm-SI and PairNorm-SCS. We use the mean aggregator and assign the sampling rate of 1 for GraphSAGE.…”

Section: Experimental Settingsmentioning

confidence: 99%

See 3 more Smart Citations

AdaGNN

Dong

Ding

Jalaian

et al. 2021

Proceedings of the 30th ACM International Conference on Information &Amp; Knowledge Management

View full text Add to dashboard Cite

Graph Neural Networks have recently become a prevailing paradigm for various high-impact graph analytical problems. Existing efforts can be mainly categorized as spectral-based and spatialbased methods. The major challenge for the former is to find an appropriate graph filter to distill discriminative information from input signals for learning. Recently, myriads of explorations are made to achieve better graph filters, e.g., Graph Convolutional Network (GCN), which leverages Chebyshev polynomial truncation to seek an approximation of graph filters and bridge these two families of methods. Nevertheless, it has been shown in recent studies that GCN and its variants are essentially employing fixed low-pass filters to perform information denoising. Thus their learning capability is rather limited and may over-smooth node representations at deeper layers. To tackle these problems, we develop a novel graph neural network framework AdaGNN with a well-designed adaptive frequency response filter. At its core, AdaGNN leverages a simple but elegant trainable filter that spans across multiple layers to capture the varying importance of different frequency components for node representation learning. The inherent differences among different feature channels are also well captured by the filter. As such, it empowers AdaGNN with stronger expressiveness and naturally alleviates the over-smoothing problem. We empirically validate the effectiveness of the proposed framework on various benchmark datasets. Theoretical analysis is also provided to show the superiority of the proposed AdaGNN. The open-source implementation of AdaGNN can be found here: https://github.com/yushundong/AdaGNN.

show abstract

Section: Resultssupporting

confidence: 88%

Section: Resultsmentioning

confidence: 67%

Section: Resultsmentioning

confidence: 99%

Section: Experimental Settingsmentioning

confidence: 99%

See 2 more Smart Citations

AdaGNN

Dong

Ding

Jalaian

et al. 2021

Proceedings of the 30th ACM International Conference on Information &Amp; Knowledge Management

View full text Add to dashboard Cite

show abstract

“…The performance degeneration is widely believed to be caused by the problem of over-smoothing [4,5,7,29,33,38,40,44,45,51]. Specifically, each graph convolutional operation tends to mix the features of connected nodes through message propagation.…”

Section: Introductionmentioning

confidence: 99%

SkipNode: On Alleviating Over-smoothing for Deep Graph Convolutional Networks

Lü¹,

Zhan²,

Lin³

et al. 2021

Preprint

View full text Add to dashboard Cite

Over-smoothing is a challenging problem, which degrades the performance of deep graph convolutional networks (GCNs). However, existing studies for alleviating the over-smoothing problem lack either generality or effectiveness. In this paper, we analyze the underlying issues behind the over-smoothing problem, i.e., featurediversity degeneration, gradient vanishing, and model weights overdecaying. Inspired by this, we propose a simple yet effective plugand-play module, SkipNode, to alleviate over-smoothing. Specifically, for each middle layer of a GCN model, SkipNode randomly (or based on node degree) selects nodes to skip the convolutional operation by directly feeding their input features to the nonlinear function. Analytically, 1) skipping the convolutional operation prevents the features from losing diversity; and 2) the "skipped" nodes enable gradients to be directly passed back, thus mitigating the gradient vanishing and model weights over-decaying issues. To demonstrate the superiority of SkipNode, we conduct extensive experiments on nine popular datasets, including both homophilic and heterophilic graphs, with different graph sizes on two typical tasks: node classification and link prediction. Specifically, 1) SkipNode has strong generalizability of being applied to various GCN-based models on different datasets and tasks; and 2) SkipNode outperforms recent state-of-the-art anti-over-smoothing plugand-play modules, i.e., DropEdge and DropNode, in different settings. Code will be made publicly available on GitHub. CCS CONCEPTS• Computing methodologies → Neural networks; Artificial intelligence; Neural networks; • Mathematics of computing → Graph algorithms.

show abstract

PDDGCN: A Parasitic Disease–Drug Association Predictor Based on Multi-view Fusion Graph Convolutional Network

Wang,

Chen,

et al. 2024

Interdiscip Sci Comput Life Sci

View full text Add to dashboard Cite

PairNorm: Tackling Oversmoothing in GNNs

Cited by 50 publications

References 7 publications

AdaGNN

AdaGNN

SkipNode: On Alleviating Over-smoothing for Deep Graph Convolutional Networks

PDDGCN: A Parasitic Disease–Drug Association Predictor Based on Multi-view Fusion Graph Convolutional Network

Contact Info

Product

Resources

About