Dark Reciprocal-Rank: Teacher-to-student Knowledge Transfer from Self-localization Model to Graph-convolutional Neural Network

Graph-based Knowledge Distillation: A survey and experimental evaluation

“…As a representative work of spatial methods, GCN [17] further simplifies graph convolution in the spectral domain by using first-order approximation, which enables graph convolution operations to be carried out in the spatial domain and greatly improves the computational efficiency of graph convolution models. Moreover, to speed up the training of graph neural networks, GNNs Graph-based Knowledge Distillation DKD methods Output layer DKWISL [18] KTG [19] DGCN [20] SPG [21] GCLN [22] Middle layer IEP [23] HKD [24] MHGD [25] IRG [26] DOD [27] HKDIFM [28] KDExplainer [29] TDD [30] DualDE [31] Constructed graph CAG [32] GKD [33] MorsE [34] BAF [35] LAD [36] GD [37] GCMT [38] GraSSNet [39] LSN [40] IntRA-KD [41] RKD [42] CC [43] SPKD [44] KCAN [45] GKD methods…”

“…For example, in DKD methods based on output layer knowledge: DKWISL [18] applies KD to relation extraction in NLP by using the KL distance metric. KTG [19] is used for image recognition applications of collaborative learning by using KL to measure the distribution difference between teachers and students, while GCLN [22] utilizes L 2 to apply KD to the visual robot localization scenarios for image semantic segmentation. Among the methods of DKD based on middlelayer knowledge: IEP [23] combines KL and L1 to apply knowledge to transfer learning and image classification on multi-task learning.…”

“…Pan et al [21] argue that the previous video description model has not clearly modeled the interaction between objects and propose a novel spatiotemporal graph network that [23] KL, L1 Multi-task learning Transfer learning, image classification HKD [24] InfoCE Knowledge distillation Image classification, knowledge transfer CAG [32] KL Graph inference Visual dialogue DKWISL [18] KL Natural language processing Relation extraction KTG [19] KL Collaborative learning Image recognition MHGD [25] KL Multi-task learning Image recognition IRG [26] Hit Knowledge distillation Image recognition DGCN [20] KL Collaborative filtering Item recommendations GKD [33] Frobenius Model compression Image classification SPG [21] KL Natural language processing Video captioning MorsE [34] L2 Meta-knowledge transfer Link prediction, question answering system GCLN [22] L2 Image semantic segmentation Vision robot self-positioning DOD [27] KL Object detection Object Detectors BAF [35] EMD Model compression Video classification LAD [36] BELU Natural language processing Machine translation GD [37] Cosine Multimodal video Motion detection, action classification GCMT [38] CE Unsupervised domain adaptation Person re-identification GraSSNet [39] MSE Knowledge transfer Saliency prediction LSN [40] KL, MSE Model compression Node classification IntRA-KD [41] MSE Model compression Road marking segmentation RKD [42] Euclidean,Huber Knowledge distillation Image classification, few-Shot Learning CC [43] KL, MSE Knowledge distillation Image classification, person re-identification SPKD [44] Frobenius Knowledge distillation Image classification, transfer learning HKDIFM [28] KL Knowledge distillation Image classification KDExplainer [29] CE, KL Interpretability Image classification TDD [30] CE, KL Interpretability Image classification DualDE [31] JSD Knowledge distillation Node classification, link prediction KCAN [45] BPR Knowledge graph Top-K Recommendation, TR Prediction explicitly uses spatiotemporal object interaction and introduce a kno...…”

“…As robot localization [22], video classification [35], event prediction [59], and road marking [41].…”