Research on Knowledge Distillation of Generative Adversarial Networks

Wang, Wei; Zhang, Baohua; Cui, Tao; Chai, Yimeng; Li, Yue

doi:10.1109/dcc50243.2021.00045

Cited by 43 publications

(59 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Wang et al [121] and Xu et al [122] combine KD with GAN. Uijlings et al [123] proposed that a group of source classes with boundary box annotations be used to revisit the knowledge transfer of training object detectors on the target class of weakly supervised training images.…”

Section: E Other Kd Methodsmentioning

confidence: 99%

Compacting Deep Neural Networks for Internet of Things: Methods and Applications

Zhang

Ying

Dai

et al. 2021

IEEE Internet Things J.

View full text Add to dashboard Cite

Deep Neural Networks (DNNs) have shown great success in completing complex tasks. However, DNNs inevitably bring high computational cost and storage consumption due to the complexity of hierarchical structures, thereby hindering their wide deployment in Internet-of-Things (IoT) devices, which have limited computational capability and storage capacity. Therefore, it is a necessity to investigate the technologies to compact DNNs. Despite tremendous advances in compacting DNNs, few surveys summarize compacting-DNNs technologies, especially for IoT applications. Hence, this paper presents a comprehensive study on compacting-DNNs technologies. We categorize compacting-DNNs technologies into three major types: 1) network model compression, 2) Knowledge Distillation (KD), 3) modification of network structures. We also elaborate on the diversity of these approaches and make side-by-side comparisons. Moreover, we discuss the applications of compacted DNNs in various IoT applications and outline future directions.

show abstract

Section: E Other Kd Methodsmentioning

confidence: 99%

Compacting Deep Neural Networks for Internet of Things: Methods and Applications

Zhang

Ying

Dai

et al. 2021

IEEE Internet Things J.

View full text Add to dashboard Cite

show abstract

“…An intuitive solution is to use the Kullback-Leibler divergence or p -loss when the knowledge falls on the soft logit [12,20] or intermediate representation [29,40]. Beyond that, Wang et al [35] utilized the adversarial training scheme in generative adversarial networks (GANs) [6] to transfer knowledge. Jang et al [16] considered meta-learning to selectively transfer knowledge.…”

Section: Related Workmentioning

confidence: 99%

Distilling a Powerful Student Model via Online Knowledge Distillation

Li¹,

Lin²,

Wang³

et al. 2021

Preprint

View full text Add to dashboard Cite

Existing online knowledge distillation approaches either adopt the student with the best performance or construct an ensemble model for better holistic performance. However, the former strategy ignores other students' information, while the latter increases the computational complexity. In this paper, we propose a novel method for online knowledge distillation, termed FFSD, which comprises two key components: Feature Fusion and Self-Distillation, towards solving the above problems in a unified framework. Different from previous works, where all students are treated equally, the proposed FFSD splits them into a student leader and a common student set. Then, the feature fusion module converts the concatenation of feature maps from all common students into a fused feature map. The fused representation is used to assist the learning of the student leader. To enable the student leader to absorb more diverse information, we design an enhancement strategy to increase the diversity among students. Besides, a self-distillation module is adopted to convert the feature map of deeper layers into a shallower one. Then, the shallower layers are encouraged to mimic the transformed feature maps of the deeper layers, which helps the students to generalize better. After training, we simply adopt the student leader, which achieves superior performance, over the common students, without increasing the storage or inference cost. Extensive experiments on CIFAR-100 and ImageNet demonstrate the superiority of our FFSD over existing works. The code is available at https://github.com/SJLeo/FFSD.

show abstract

“…They validated their findings by running on MNIST dataset and, JFT dataset by google and other speech recognition tasks. Since then, knowledge distillation has progressed a lot, and adversarial methods [17,18] also have utilized for modelling knowledge transfer between teacher and student. After this study, extensive research has conducted on knowledge distillation.…”

Section: Related Studymentioning

confidence: 99%

Exploring Knowledge Distillation of a Deep Neural Network for Multi-Script identification

Dastidar

Dutta

Das

et al. 2021

Preprint

View full text Add to dashboard Cite

Multi-lingual script identification is a difficult task consisting of different language with complex backgrounds in scene text images. According to the current research scenario, deep neural networks are employed as teacher models to train a smaller student network by utilizing the teacher model's predictions. This process is known as dark knowledge transfer. It has been quite successful in many domains where the final result obtained is unachievable through directly training the student network with a simple architecture. In this paper, we explore dark knowledge transfer approach using long short-term memory(LSTM) and CNN based assistant model and various deep neural networks as the teacher model, with a simple CNN based student network, in this domain of multi-script identification from natural scene text images. We explore the performance of different teacher models and their ability to transfer knowledge to a student network. Although the small student network's limited size, our approach obtains satisfactory results on a well-known script identification dataset CVSI-2015.

show abstract

Research on Knowledge Distillation of Generative Adversarial Networks

Cited by 43 publications

References 0 publications

Compacting Deep Neural Networks for Internet of Things: Methods and Applications

Compacting Deep Neural Networks for Internet of Things: Methods and Applications

Distilling a Powerful Student Model via Online Knowledge Distillation

Exploring Knowledge Distillation of a Deep Neural Network for Multi-Script identification

Contact Info

Product

Resources

About