TinyGNN: Learning Efficient Graph Neural Networks

Yan, Bencheng; Wang, Chaokun; Guo, Gaoyang; Lou, Yunkai

doi:10.1145/3394486.3403236

Cited by 45 publications

(36 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Two other kinds of methods considered as inference acceleration are GNN-to-GNN KD like TinyGNN (Yan et al, 2020) and Graph Augmented-MLPs (GA-MLPs) like SGC (Wu et al, 2019) or SIGN (Frasca et al, 2020). Inference of GNN-to-GNN KD is likely to be slower than a GNN-Li with the same i as the student, since there will usually be overheads introduced by some extra modules like the Peer-Aware Module (PAM) in TinyGNN.…”

Section: How Do Glnns Compare To Other Inference Acceleration Methods?mentioning

confidence: 99%

“…Inference acceleration schemes have been proposed by hardware improvements (Chen et al, 2016;Judd et al, 2016) and algorithmic improvements through pruning (Han et al, 2015), quantization (Gupta et al, 2015), and KD (Hinton et al, 2015). Specifically for GNNs, pruning (Zhou et al, 2021) and quantizing GNN parameters (Tailor et al, 2021;Zhao et al, 2020), or distilling to smaller GNNs (Yang et al, 2021b;Yan et al, 2020;Yang et al, 2021a) have been studied. These approaches speed up GNN inference to a certain extent but do not eliminate the neighborhood-fetching latency.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Graph-less Neural Networks: Teaching Old MLPs New Tricks via Distillation

Zhang¹,

Liu²,

Sun³

et al. 2021

Preprint

View full text Add to dashboard Cite

Graph Neural Networks (GNNs) have recently become popular for graph machine learning and have shown great results on wide node classification tasks. Yet, GNNs are less popular for practical deployments in the industry owing to their scalability challenges incurred by data dependency. Namely, GNN inference depends on neighbor nodes multiple hops away from the target, and fetching these nodes burdens latency-constrained applications. Existing inference acceleration methods like pruning and quantization can speed up GNNs to some extent by reducing Multiplication-and-ACcumulation (MAC) operations. However, their improvements are limited given the data dependency is not resolved. Conversely, multi-layer perceptrons (MLPs) have no dependency on graph data and infer much faster than GNNs, even though they are less accurate than GNNs for node classification in general. Motivated by these complementary strengths and weaknesses, we bring GNNs and MLPs together via knowledge distillation (KD). Our work shows that the performance of MLPs can be improved by large margins with GNN KD. We call the distilled MLPs Graph-less Neural Networks (GLNNs) as they have no inference graph dependency. We show that GLNNs with competitive performance infer faster than GNNs by 146×-273× and faster than other acceleration methods by 14×-27×. Meanwhile, under a production setting involving both transductive and inductive predictions across 7 datasets, GLNN accuracies improve over stand-alone MLPs by 12.36% on average and match GNNs on 6/7 datasets. A comprehensive analysis of GLNN shows when and why GLNNs can achieve competitive results to GNNs and suggests GLNN as a handy choice for latency-constrained applications.

show abstract

Section: How Do Glnns Compare To Other Inference Acceleration Methods?mentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Graph-less Neural Networks: Teaching Old MLPs New Tricks via Distillation

Zhang¹,

Liu²,

Sun³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Therefore, VGAER allows us to use any GNN model or plug and play module in the encoding stage, such as: GraphSAGE (Hamilton, Ying, and Leskovec 2017), which realizes inductive learning through neighborhood sampling encoding and greatly reduces the complexity of the algorithm. We can see the extension of VGAER on GraphSAGE in the next section for the scalability; GAT (Veličković et al 2017), which enables VGAER to capture the weights between different node modules; PAM (Yan et al 2020), which is actually a plug and play module with linear complexity, and faster incremental community detection can be achieved on the basis of Graph-SAGE inductive learning; GIN (Xu et al 2018), which introduce a to realize the injectivity of aggregation, thereby achieving a more powerful aggregation function.…”

Section: Referencementioning

confidence: 99%

VGAER: Graph Neural Network Reconstruction based Community Detection

Qiu¹,

Huang²,

Xu³

et al. 2022

Preprint

View full text Add to dashboard Cite

Community detection is a fundamental and important issue in network science, but there are only a few community detection algorithms based on graph neural networks, among which unsupervised algorithms are almost blank. By fusing the high-order modularity information with network features, this paper proposes a Variational Graph AutoEncoder Reconstruction based community detection VGAER for the first time, and gives its non-probabilistic version. They do not need any prior information. We have carefully designed corresponding input features, decoder, and downstream tasks based on the community detection task and these designs are concise, natural, and perform well (NMI values under our design are improved by 59.1% -565.9%). Based on a series of experiments with wide range of datasets and advanced methods, VGAER has achieved superior performance and shows strong competitiveness and potential with a simpler design. Finally, we report the results of algorithm convergence analysis and t-SNE visualization, which clearly depicted the stable performance and powerful network modularity ability of VGAER. Our codes are available at https://github.com/qcydm/VGAER.

show abstract

“…Such operations indirectly capture as much deep neighborhood information as possible in the case of one layer. Let the v have K neighbors b l i K i=1 , the neighbor nodes after passing through the PAM are expressed as [16]:…”

Section: Algorithm 2 Feature Alignmentmentioning

confidence: 99%

Fast Community Detection based on Graph Autoencoder Reconstruction

Qiu¹,

Huang²,

Xu³

et al. 2022

Preprint

View full text Add to dashboard Cite

With the rapid development of big data, how to efficiently and accurately discover tight community structures in large-scale networks for knowledge discovery has attracted more and more attention. In this paper, a community detection framework based on Graph AutoEncoder Reconstruction (noted as GAER) is proposed for the first time. GAER is a highly scalable framework which does not require any prior information. We decompose the graph autoencoder-based one-step encoding into the two-stage encoding framework to adapt to the real-world big data system by reducing complexity from the original O(N 2 ) to O(N ). At the same time, based on the advantages of GAER support module plug-and-play configuration and incremental community detection, we further propose a peer awareness based module for real-time large graphs, which can realize the new nodes community detection at a faster speed, and accelerate model inference with the 6.15×-14.03× speed. Finally, we apply the GAER on multiple real-world datasets, including some largescale networks. The experimental result verified that GAER has achieved the superior performance on almost all networks.

show abstract

TinyGNN: Learning Efficient Graph Neural Networks

Cited by 45 publications

References 10 publications

Graph-less Neural Networks: Teaching Old MLPs New Tricks via Distillation

Graph-less Neural Networks: Teaching Old MLPs New Tricks via Distillation

VGAER: Graph Neural Network Reconstruction based Community Detection

Fast Community Detection based on Graph Autoencoder Reconstruction

Contact Info

Product

Resources

About