Data-Free Learning of Student Networks

Chen, Hanting; Wang, Yunhe; Xu, Chang; Yang, Zhaohui; Liu, Chuanjian; Shi, Boxin; Xu, Chunjing; Xu, Chao; Tian, Qi

doi:10.1109/iccv.2019.00361

Cited by 266 publications

(270 citation statements)

References 25 publications

Supporting

Mentioning

245

Contrasting

Order By: Relevance

“…Nayak et al [26] propose to generate training data for student using data impressions by modeling the softmax space using Dirichlet distribution. Recently proposed methods directly generate synthesized training data for the student network using adversarial learning [27] [28][29] [30] or deepdream data propagation [31] [32]. Although these algorithms can train the student networks without any available training data of the teacher, none has reported performance comparable to supervised methods.…”

Section: Data-free Knowledge Distillationmentioning

confidence: 99%

Industrial Cyber-Physical Systems-Based Cloud IoT Edge for Federated Heterogeneous Distillation

Wang

Yang

Papanastasiou

et al. 2021

IEEE Trans. Ind. Inf.

View full text Add to dashboard Cite

Deep convoloutional networks have been widely deployed in modern cyber-physical systems performing different visual classification tasks. As the fog and edge devices have different computing capacity and perform different sub-tasks, models trained for one device may not be deployable on another. Knowledge distillation technique can effectively compress well trained convolutional neural networks (CNN) into lightweight models suitable to different devices. However, due to privacy issue and transmission cost, manually annotated data for training the deep learning models are usually gradually collected and archived in different sites. Simply training a model on powerful cloud servers and compressing them for particular edge devices failed to use the distributed data stored at different sites. This offline training approach is also inefficient to deal with new data collected from the edge devices. To overcome these obstacles, we propose the heterogeneous brain storming (HBS) method for object recognition tasks in real-world IoT scenarios. Our method enables flexible bidirectional federated learning of heterogeneous models trained on distributed datasets with a new "brain storming" mechanism and optimizable temperature parameters. In our comparison experiments, this heterogeneous brain storming method outperformed multiple state-of-the-art single-model compression methods, as well as the newest multinetwork knowledge distillation methods with both homogeneous and heterogeneous classifiers. The ablation experiment results proved that the trainable temperature parameter into the conventional knowledge distillation loss can effectively ease the learning process of student networks in different methods. To the best of

show abstract

Section: Data-free Knowledge Distillationmentioning

confidence: 99%

Industrial Cyber-Physical Systems-Based Cloud IoT Edge for Federated Heterogeneous Distillation

Wang

Yang

Papanastasiou

et al. 2021

IEEE Trans. Ind. Inf.

View full text Add to dashboard Cite

show abstract

“…In particular, this task could be regarded as a generative adversarial task if we take D mod , G sp as discriminator and generator in generative adversarial network respectively. In [22], a generator is trained to map the gaussian noise to an image of handwritten digits and only a well-trained digits classification module called teacher network is used. There are some similarities between that work with ours.…”

Section: ) Signal Processing Modulementioning

confidence: 99%

“…There are some similarities between that work with ours. Based on our specific task and the experience in [22], we formulate our loss function as…”

Section: ) Signal Processing Modulementioning

confidence: 99%

Dive Into Deep Learning Based Automatic Modulation Classification: A Disentangled Approach

Shang

et al. 2020

IEEE Access

View full text Add to dashboard Cite

Recently, deep learning (DL) based automatic modulation classification (AMC) has received much attention. Various network structures with higher complexity are utilized to boost the performance of classification model. We divide the issue of AMC into two objectives and propose a disentangled approach with a signal processing module. Unlike popular end-to-end training strategy, we first consider a simple model with much fewer trainable parameters to learn accurate modulation features for classification. Then a U-net based signal processing module using a specially designed function is introduced to transfer the knowledge stored in classification module. We compare the performance of the proposed method with several baseline models on two well known datasets. Experimental results demonstrate that the proposed method gives superior performance with lower computational complexity compared with other methods. Furthermore, we also verify the feasibility and huge potential of the knowledge transferring in the field of wireless communications. INDEX TERMS Automatic modulation classification, deep learning, knowledge transfer.

show abstract

“…Among others, Hanting et al proposed a knowledge distillation method that does not require the participation of the original training data during model compression. By combining the ideas of the generative adversarial network (GAN) [47] and knowledge distillation, the student network is able to effectively learn the performance of the teacher network [48]. In [49], the authors proposed a knowledge distillation method based on the correlation between instances.…”

Section: Knowledge Distillationmentioning

confidence: 99%

Compressing Deep Networks by Neuron Agglomerative Clustering

Wang

Liu

et al. 2020

Sensors

View full text Add to dashboard Cite

In recent years, deep learning models have achieved remarkable successes in various applications, such as pattern recognition, computer vision, and signal processing. However, high-performance deep architectures are often accompanied by a large storage space and long computational time, which make it difficult to fully exploit many deep neural networks (DNNs), especially in scenarios in which computing resources are limited. In this paper, to tackle this problem, we introduce a method for compressing the structure and parameters of DNNs based on neuron agglomerative clustering (NAC). Specifically, we utilize the agglomerative clustering algorithm to find similar neurons, while these similar neurons and the connections linked to them are then agglomerated together. Using NAC, the number of parameters and the storage space of DNNs are greatly reduced, without the support of an extra library or hardware. Extensive experiments demonstrate that NAC is very effective for the neuron agglomeration of both the fully connected and convolutional layers, which are common building blocks of DNNs, delivering similar or even higher network accuracy. Specifically, on the benchmark CIFAR-10 and CIFAR-100 datasets, using NAC to compress the parameters of the original VGGNet by 92.96% and 81.10%, respectively, the compact network obtained still outperforms the original networks.

show abstract

Data-Free Learning of Student Networks

Cited by 266 publications

References 25 publications

Industrial Cyber-Physical Systems-Based Cloud IoT Edge for Federated Heterogeneous Distillation

Industrial Cyber-Physical Systems-Based Cloud IoT Edge for Federated Heterogeneous Distillation

Dive Into Deep Learning Based Automatic Modulation Classification: A Disentangled Approach

Compressing Deep Networks by Neuron Agglomerative Clustering

Contact Info

Product

Resources

About