EGAD: Evolving Graph Representation Learning with Self-Attention and Knowledge Distillation for Live Video Streaming Events

Antaris, Stefanos; Rafailidis, Dimitrios; Girdzijauskas, Šarūnas

doi:10.1109/bigdata50022.2020.9378219

Cited by 3 publications

(3 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…GFKD [46] RDD [47] GKD [48] GLNN [49] Distill2Vec [50] MT-GCN [51] TinyGNN [52] GLocalKD [53] SCR [54] ROD [55] EGNN [56] Middle layer LWC-KD [57] MustaD [58] EGAD [59] AGNN [60] Cold Brew [61] PGD [62] OAD [63] CKD [64] BGNN [65] EGSC [66] HSKDM [67] Constructed graph GRL [68] GFL [69] HGKT [70] CPF [71] LSP [16] scGCN [72] MetaHG [73] G-CRD [74] HIRE [75] SKD methods…”

Section: Output Layermentioning

confidence: 99%

“…Additionally, the middle layer knowledge of GKD is also used for other tasks. For example, Anantis et al [59] first utilize EGAD for dynamic graph representation learning, introducing a weighted selfattention mechanism between continuous dynamic graph convolutional networks to capture the evolution of realtime video stream event graphs. Bahri et al [65] introduce a binary graph neural network named BGNN based on XNOR-Net++ and knowledge distillation and also explore the influence on the image classification performance of binary graph neural networks by studying various strategies and design decisions.…”

Section: Middle Layer Knowledgementioning

confidence: 99%

“…As robot localization [22], video classification [35], event prediction [59], and road marking [41].…”

Section: Computer Visionmentioning

confidence: 99%

See 2 more Smart Citations

Graph-based Knowledge Distillation: A survey and experimental evaluation

Liu¹,

Tongya²,

Zhang³

et al. 2023

Preprint

View full text Add to dashboard Cite

Graph data, such as citation networks, social networks, and transportation networks, are prevalent in the real world. Graph neural networks (GNNs) have gained widespread attention for their robust expressiveness and exceptional performance in various graph analysis applications. However, the efficacy of GNNs is heavily reliant on sufficient data labels and complex network models, with the former being challenging to obtain and the latter requiring expensive computational resources. To address the labeled data scarcity and high complexity of GNNs, Knowledge Distillation (KD) has been introduced to enhance existing GNNs. This technique involves transferring the soft-label supervision of the large teacher model to the small student model while maintaining prediction performance. Transferring the KD technique to graph data and graph-based knowledge is a major challenge. This survey offers a comprehensive overview of Graph-based Knowledge Distillation methods, systematically categorizing and summarizing them while discussing their limitations and future directions. This paper first introduces the background of graph and KD. It then provides a comprehensive summary of three types of Graph-based Knowledge Distillation methods, namely Graph-based Knowledge Distillation for deep neural networks (DKD), Graph-based Knowledge Distillation for GNNs (GKD), and Self-Knowledge Distillation based Graph-based Knowledge Distillation (SKD). Each type of method is further divided into knowledge distillation methods based on the output layer, middle layer, and constructed graph. Subsequently, various graph-based knowledge distillation algorithms' ideas are analyzed and compared, concluding with the advantages and disadvantages of each algorithm supported by experimental results. In addition, the applications of graph-based knowledge distillation in computer vision, natural language processing, recommendation systems, and other fields are listed. Finally, the development of graph-based knowledge distillation is summarized and prospectively discussed. We have also released related resources at https://github.com/liujing1023/Graph-based-Knowledge-Distillation.

show abstract

Section: Output Layermentioning

confidence: 99%

Section: Middle Layer Knowledgementioning

confidence: 99%

See 1 more Smart Citation

Graph-based Knowledge Distillation: A survey and experimental evaluation

Liu¹,

Tongya²,

Zhang³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

Knowledge distillation on neural networks for evolving graphs

Antaris

Rafailidis

Girdzijauskas

2021

Soc. Netw. Anal. Min.

View full text Add to dashboard Cite

Graph representation learning on dynamic graphs has become an important task on several real-world applications, such as recommender systems, email spam detection, and so on. To efficiently capture the evolution of a graph, representation learning approaches employ deep neural networks, with large amount of parameters to train. Due to the large model size, such approaches have high online inference latency. As a consequence, such models are challenging to deploy to an industrial setting with vast number of users/nodes. In this study, we propose DynGKD, a distillation strategy to transfer the knowledge from a large teacher model to a small student model with low inference latency, while achieving high prediction accuracy. We first study different distillation loss functions to separately train the student model with various types of information from the teacher model. In addition, we propose a hybrid distillation strategy for evolving graph representation learning to combine the teacher’s different types of information. Our experiments with five publicly available datasets demonstrate the superiority of our proposed model against several baselines, with average relative drop $$40.60\%$$ 40.60 % in terms of RMSE in the link prediction task. Moreover, our DynGKD model achieves a compression ratio of 21:100, accelerating the inference latency with a speed up factor $$\times 30$$ × 30 , when compared with the teacher model. For reproduction purposes, we make our datasets and implementation publicly available at https://github.com/stefanosantaris/DynGKD.

show abstract

Distill2Vec: Dynamic Graph Representation Learning with Knowledge Distillation

Antaris

Rafailidis

2020

2020 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM)

View full text Add to dashboard Cite

EGAD: Evolving Graph Representation Learning with Self-Attention and Knowledge Distillation for Live Video Streaming Events

Cited by 3 publications

References 27 publications

Graph-based Knowledge Distillation: A survey and experimental evaluation

Graph-based Knowledge Distillation: A survey and experimental evaluation

Knowledge distillation on neural networks for evolving graphs

Distill2Vec: Dynamic Graph Representation Learning with Knowledge Distillation

Contact Info

Product

Resources

About