HIRE: Distilling high-order relational knowledge from heterogeneous graph neural networks

Liu, Jing; Tongya, Zheng,; Hao, Qinfen

doi:10.1016/j.neucom.2022.08.022

Cited by 4 publications

(6 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…GFKD [46] RDD [47] GKD [48] GLNN [49] Distill2Vec [50] MT-GCN [51] TinyGNN [52] GLocalKD [53] SCR [54] ROD [55] EGNN [56] Middle layer LWC-KD [57] MustaD [58] EGAD [59] AGNN [60] Cold Brew [61] PGD [62] OAD [63] CKD [64] BGNN [65] EGSC [66] HSKDM [67] Constructed graph GRL [68] GFL [69] HGKT [70] CPF [71] LSP [16] scGCN [72] MetaHG [73] G-CRD [74] HIRE [75] SKD methods…”

Section: Output Layermentioning

confidence: 99%

“…Insight and strength: Compared with the DKD method, the GKD differs most: GNN is a powerful tool for modeling graphs, which can be directly distilled in the middle/output layer and transfer the topological knowledge between graph nodes to the student model. To further explore the relationship between local nodes in the feature space, a lot of efforts have been made to construct the relational graph between nodes in the middle convolutional layer to extract the correlation knowledge between, such as LSP [16], HIRE [75], etc. The successful application of knowledge distillation in GNNs has attracted widespread attention from academia and industry.…”

Section: Graph-based Knowledge Distillation For Graph Neural Networkmentioning

confidence: 99%

“…This section delineates Graph-based Knowledge Distillation for Graph Neural Networks (GKD) methods in detail, as shown in TABLE 3 below Differences: TABLE 3 shows the differences between various types of GKD. Output layer-based methods using KL divergence are GFKD [46], GLNN [49], Distill2Vec [50], and MT-GCN [51]; middle layer-based methods using KL divergence are MustaD [58], OAD [63], BGNN [65], and HSKDM [67]; constructed graph-based methods using KL divergence are CPF [71], LSP [16], MetaHG [73], and HIRE [75]. Similarly, output layer-based methods using MSE are RDD [47] and SCR [54]; middle layer-based methods using MSE are LWC-KD [57], AGNN [60], Cold Brew [61], and EGSC [66].…”

Section: Graph-based Knowledge Distillation For Graph Neural Networkmentioning

confidence: 99%

“…Similarly, output layer-based methods using MSE are RDD [47] and SCR [54]; middle layer-based methods using MSE are LWC-KD [57], AGNN [60], Cold Brew [61], and EGSC [66]. Finally, HIRE [75] uses the constructed graph knowledge of a relational metric types to measure the differences between teacher and student models.…”

Section: Graph-based Knowledge Distillation For Graph Neural Networkmentioning

confidence: 99%

“…To further enrich and provide more general knowledge, GNNs will further learn the topological structure and node relationship information of the teacher model with the help of constructed graphs [16,[68][69][70][71][72][73][74][75], so as to deeply explore the knowledge contained in the teacher model.…”

Section: Constructed Graph Knowledgementioning

confidence: 99%

See 4 more Smart Citations

Graph-based Knowledge Distillation: A survey and experimental evaluation

Liu¹,

Tongya²,

Zhang³

et al. 2023

Preprint

View full text Add to dashboard Cite

Graph data, such as citation networks, social networks, and transportation networks, are prevalent in the real world. Graph neural networks (GNNs) have gained widespread attention for their robust expressiveness and exceptional performance in various graph analysis applications. However, the efficacy of GNNs is heavily reliant on sufficient data labels and complex network models, with the former being challenging to obtain and the latter requiring expensive computational resources. To address the labeled data scarcity and high complexity of GNNs, Knowledge Distillation (KD) has been introduced to enhance existing GNNs. This technique involves transferring the soft-label supervision of the large teacher model to the small student model while maintaining prediction performance. Transferring the KD technique to graph data and graph-based knowledge is a major challenge. This survey offers a comprehensive overview of Graph-based Knowledge Distillation methods, systematically categorizing and summarizing them while discussing their limitations and future directions. This paper first introduces the background of graph and KD. It then provides a comprehensive summary of three types of Graph-based Knowledge Distillation methods, namely Graph-based Knowledge Distillation for deep neural networks (DKD), Graph-based Knowledge Distillation for GNNs (GKD), and Self-Knowledge Distillation based Graph-based Knowledge Distillation (SKD). Each type of method is further divided into knowledge distillation methods based on the output layer, middle layer, and constructed graph. Subsequently, various graph-based knowledge distillation algorithms' ideas are analyzed and compared, concluding with the advantages and disadvantages of each algorithm supported by experimental results. In addition, the applications of graph-based knowledge distillation in computer vision, natural language processing, recommendation systems, and other fields are listed. Finally, the development of graph-based knowledge distillation is summarized and prospectively discussed. We have also released related resources at https://github.com/liujing1023/Graph-based-Knowledge-Distillation.

show abstract