Distilling Self-Knowledge From Contrastive Links to Classify Graph Nodes Without Passing Messages

Luo, Yi; Chen, Aiguo; Yan, Ke; Tian, Ling

doi:10.48550/arxiv.2106.08541

Cited by 5 publications

(9 citation statements)

References 12 publications

(12 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…PubMed Reddit Ogbn-products GCN [15] 87.70% 93.3%* 75.64%* GAT [24] 86.80% − − LinkDist [17] 89.58%* − − GraphSAINT [29] 88.50% 96.6%* 79.08%* CAGNET [23] 87.60% 93.73% 75.36% CES-GCN [13] N/A 92.4%* N/A BDS-GCN [25] N/A 97.07%* 79.44%* PPSGCN 90.10% 96.05% 79.15%…”

Section: Methodsmentioning

confidence: 99%

See 1 more Smart Citation

PPSGCN: A Privacy-Preserving Subgraph Sampling Based Distributed GCN Training Method

Zhang¹,

Luo²,

Feng³

et al. 2021

Preprint

View full text Add to dashboard Cite

Graph convolutional networks (GCNs) have been widely adopted for graph representation learning and achieved impressive performance. For larger graphs stored separately on different clients, distributed GCN training algorithms were proposed to improve efficiency and scalability. However, existing methods directly exchange node features between different clients, which results in data privacy leakage. Federated learning was incorporated in graph learning to tackle data privacy, while they suffer from severe performance drop due to non-iid data distribution. Besides, these approaches generally involve heavy communication and memory overhead during the training process. In light of these problems, we propose a Privacy-Preserving Subgraph sampling based distributed GCN training method (PPSGCN), which preserves data privacy and significantly cuts back on communication and memory overhead. Specifically, PPSGCN employs a star-topology client-server system. We firstly sample a local node subset in each client to form a global subgraph, which greatly reduces communication and memory costs. We then conduct local computation on each client with features or gradients of the sampled nodes. Finally, all clients securely communicate with the central server with homomorphic encryption to combine local results while preserving data privacy. Compared with federated graph learning methods, our PPSGCN model is trained on a global graph to avoid the negative impact of local data distribution. We prove that our PPSGCN algorithm would converge to a local optimum with probability 1. Experiment results on three prevalent benchmarks demonstrate that our algorithm significantly reduces communication and memory overhead while maintaining desirable performance. Further studies not only demonstrate the fast convergence of PPSGCN, but discuss the trade-off between communication and local computation cost as well.

show abstract

Section: Methodsmentioning

confidence: 99%

“…• GAT [24], which introduces the attention mechanism to graph neural networks. • LinkDist [17], which substitutes message-passing neighbour aggregation with MLP in graph neural networks. • GraphSAINT [29], which proposes an inductive GCN training method with subgraph sampling and normalization.…”

Section: Experimental Set-upmentioning

confidence: 99%

PPSGCN: A Privacy-Preserving Subgraph Sampling Based Distributed GCN Training Method

Zhang¹,

Luo²,

Feng³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…45 LinkDist series (LinkDistMLP, CoLinkDist, and LinkDist) extract useful features by distilling self-knowledge from associated couple nodes. 46 3ference analyzes the transition patterns of node labels on the graph. 47 The performances of these SOTA approaches are improved.…”

Section: Comparison With More Sota Approachesmentioning

confidence: 99%

Graph Decipher: A transparent dual‐attention graph neural network to understand the message‐passing mechanism for the node classification

Pang¹,

Liu²

2022

Int J of Intelligent Sys

View full text Add to dashboard Cite

Graph neural networks (GNNs) can be effectively applied to solve many real-world problems across widely diverse fields. Their success is inseparable from the message-passing mechanisms evolving over the years. However, current mechanisms treat all node features equally at the macro-level (node-level), and the optimal aggregation method has not yet been explored. In this paper, we propose a new GNN called Graph Decipher (GD), which transparentizes the message flows of node features from micro-level (feature-level) to global-level and boosts the performance on node classification tasks.Besides, to reduce the computational burden caused by investigating message-passing, only the relevant representative node attributes are extracted by graph feature filters, allowing calculations to be performed in a category-oriented manner. Experiments on 10 node classification data sets show that GD achieves state-ofthe-art performance while imposing a substantially lower computational cost. Additionally, since GD has the ability to explore the representative node attributes

show abstract

“…For example, Graph-MLP (Hu et al, 2021) designs a neighborhood contrastive loss to bridge the gap between GNNs and MLPs by implicitly utilizing the adjacency information. Instead, LinkDist (Luo et al, 2021) directly distills knowledge from connected node pairs into MLPs without message passing. Despite their great progress, these methods still cannot match the state-of-the-art GNNs in terms of classification performance due to the lack of modeling the graph topology.…”

Section: Related Workmentioning

confidence: 99%

“…As illustrated in Fig. 1(c), the MLP-based models, such as Graph-MLP (Hu et al, 2021) and LinkDist (Luo et al, 2021), are faster in inference but with much poorer performance compared to GNNs. There are two main branches of existing approaches to connect these two worlds.…”

Section: Introductionmentioning

confidence: 99%

Teaching Yourself: Graph Self-Distillation on Neighborhood for Node Classification

Wu¹,

Xia²,

Hua³

et al. 2022

Preprint

View full text Add to dashboard Cite

Recent years have witnessed great success in handling graph-related tasks with Graph Neural Networks (GNNs). Despite their great academic success, Multi-Layer Perceptrons (MLPs) remain the primary workhorse for practical industrial applications. One reason for this academic-industrial gap is the neighborhoodfetching latency incurred by data dependency in GNNs, which make it hard to deploy for latency-sensitive applications that require fast inference. Conversely, without involving any feature aggregation, MLPs have no data dependency and infer much faster than GNNs, but their performance is less competitive. Motivated by these complementary strengths and weaknesses, we propose a Graph Self-Distillation on Neighborhood (GSDN) framework to reduce the gap between GNNs and MLPs. Specifically, the GSDN framework is based purely on MLPs, where structural information is only implicitly used as prior to guide knowledge self-distillation between the neighborhood and the target, substituting the explicit neighborhood information propagation as in GNNs. As a result, GSDN enjoys the benefits of graph topology-awareness in training but has no data dependency in inference. Extensive experiments have shown that the performance of vanilla MLPs can be greatly improved with self-distillation, e.g., GSDN improves over standalone MLPs by 15.54% on average and outperforms the state-of-the-art GNNs on six datasets. Regarding inference speed, GSDN infers 75×-89× faster than existing GNNs and 16×-25× faster than other inference acceleration methods.

show abstract

Distilling Self-Knowledge From Contrastive Links to Classify Graph Nodes Without Passing Messages

Cited by 5 publications

References 12 publications

PPSGCN: A Privacy-Preserving Subgraph Sampling Based Distributed GCN Training Method

PPSGCN: A Privacy-Preserving Subgraph Sampling Based Distributed GCN Training Method

Graph Decipher: A transparent dual‐attention graph neural network to understand the message‐passing mechanism for the node classification

Teaching Yourself: Graph Self-Distillation on Neighborhood for Node Classification

Contact Info

Product

Resources

About