Dreaming to Distill: Data-Free Knowledge Transfer via DeepInversion

Yin, Hongxu; Molchanov, Pavlo; Álvarez, Jose M.; Li, Zhizhong; Mallya, Arun; Hoiem, Derek; Jha, Niraj K.; Kautz, Jan

doi:10.1109/cvpr42600.2020.00874

Cited by 295 publications

(160 citation statements)

References 29 publications

Supporting

Mentioning

130

Contrasting

Order By: Relevance

“…Some researchers have looked at optimizing an image map to maximize the activations of an individual or set of neurons [140]. Yin et al [141] generate images from activations for the sake of providing a knowledge distillation [142] set for compressing the network. Riberio et al [143] propose the LIME framework to sample around a data point to provide analysis of what caused the prediction.…”

Section: Interpretabilitymentioning

confidence: 99%

Deep Learning applications for COVID-19

2021

View full text Add to dashboard Cite

This survey explores how Deep Learning has battled the COVID-19 pandemic and provides directions for future research on COVID-19. We cover Deep Learning applications in Natural Language Processing, Computer Vision, Life Sciences, and Epidemiology. We describe how each of these applications vary with the availability of big data and how learning tasks are constructed. We begin by evaluating the current state of Deep Learning and conclude with key limitations of Deep Learning for COVID-19 applications. These limitations include Interpretability, Generalization Metrics, Learning from Limited Labeled Data, and Data Privacy. Natural Language Processing applications include mining COVID-19 research for Information Retrieval and Question Answering, as well as Misinformation Detection, and Public Sentiment Analysis. Computer Vision applications cover Medical Image Analysis, Ambient Intelligence, and Vision-based Robotics. Within Life Sciences, our survey looks at how Deep Learning can be applied to Precision Diagnostics, Protein Structure Prediction, and Drug Repurposing. Deep Learning has additionally been utilized in Spread Forecasting for Epidemiology. Our literature review has found many examples of Deep Learning systems to fight COVID-19. We hope that this survey will help accelerate the use of Deep Learning for COVID-19 research.

show abstract

Section: Interpretabilitymentioning

confidence: 99%

Deep Learning applications for COVID-19

2021

View full text Add to dashboard Cite

show abstract

“…1 A classification problem over 10 classes is aimed for each dataset. We compare our RDSKD method with DAFL [9], MSKD where the denominator of L DS is replaced by the mode seeking loss in [10], data-free KD via DeepInversion (DeepI) [18], and zero-shot KD (ZSKD) [19]. For DeepI and ZSKD, image samples are directly generated from the teacher without a separate generator.…”

Section: Models and Parameter Settingsmentioning

confidence: 99%

“…It is also more challenging than training the generator and student simultaneously. Most works in the literature also introduced additional hyperparameters [9,17,18], requiring more efforts for parameter tuning. Leveraging and extending these preceding works, we develop RDSKD that is free from original data or meta-data while not requiring to fine-tune hyperparameters, which is in stark contrast to DAFL [9].…”

Section: Introductionmentioning

confidence: 99%

Robustness and Diversity Seeking Data-Free Knowledge Distillation

Han

Park

Wang

et al. 2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

Knowledge distillation (KD) has enabled remarkable progress in model compression and knowledge transfer. However, KD requires a large volume of original data or their representation statistics that are not usually available in practice. Datafree KD has recently been proposed to resolve this problem, wherein teacher and student models are fed by a synthetic sample generator trained from the teacher. Nonetheless, existing data-free KD methods rely on fine-tuning of weights to balance multiple losses, and ignore the diversity of generated samples, resulting in limited accuracy and robustness. To overcome this challenge, we propose robustness and diversity seeking data-free KD (RDSKD) in this paper. The generator loss function is crafted to produce samples with high authenticity, class diversity, and inter-sample diversity. Without real data, the objectives of seeking high sample authenticity and class diversity often conflict with each other, causing frequent loss fluctuations. We mitigate this by exponentially penalizing loss increments. With MNIST, CIFAR-10, and SVHN datasets, our experiments show that RDSKD achieves higher accuracy with more robustness over different hyperparameter settings, compared to other data-free KD methods such as DAFL, MSKD, ZSKD, and DeepInversion.

show abstract

“…Specifically for super-resolution, generative adversarial network has been proven to be an advanced algorithm that can extract semantic features of images and improve the resolution of low-quality data. 27 The semantic features of medical images are very important for the identification of disease types. Therefore, compared with traditional algorithms of coding and filter, the generative adversarial network can process CT images better.…”

Section: Introductionmentioning

confidence: 99%

Artificial Intelligence Clinicians Can Use Chest Computed Tomography Technology to Automatically Diagnose Coronavirus Disease 2019 (COVID-19) Pneumonia and Enhance Low-Quality Images

Zhang¹,

Chen²,

Li³

et al. 2021

IDR

View full text Add to dashboard Cite

Purpose Nowadays, the number of patients with COVID-19 pneumonia worldwide is still increasing. The clinical diagnosis of COVID-19 pneumonia faces challenges, such as the difficulty to perform RT-PCR tests in real time, the lack of experienced radiologists, clinical low-quality images, and the similarity of imaging features of community-acquired pneumonia and COVID-19. Therefore, we proposed an artificial intelligence model GARCD that uses chest CT images to assist in the diagnosis of COVID-19 in real time. It can show better diagnostic performance even facing low-quality CT images. Methods We used 14,129 CT images from 104 patients. A total of 12,929 samples were used to build artificial intelligence models, and 1200 samples were used to test its performance. The image quality improvement module is based on the generative adversarial structure. It improves the quality of the input image under the joint drive of feature loss and content loss. The enhanced image is sent to the disease diagnosis model based on residual convolutional network. It automatically extracts the semantic features of the image and then infers the probability that the sample belongs to COVID-19. The ROC curve is used to evaluate the performance of the model. Results This model can effectively enhance the low-quality image and make the image that is difficult to be recognized become recognizable. The model proposed in this paper reached 97.8% AUC, 96.97% sensitivity and 91.16% specificity in an independent test set. ResNet, GADCD, CNN, and DenseNet achieved 80.9%, 97.3%, 70.7% and 85.7% AUC in the same test set, respectively. By comparing the performance with related works, it is proved that the model proposed has stronger clinical usability. Conclusion The method proposed can effectively assist doctors in real-time detection of suspected cases of COVID-19 pneumonia even faces unclear image. It can quickly isolate patients in a targeted manner, which is of positive significance for preventing the further spread of COVID-19 pneumonia.

show abstract

Dreaming to Distill: Data-Free Knowledge Transfer via DeepInversion

Cited by 295 publications

References 29 publications

Deep Learning applications for COVID-19

Deep Learning applications for COVID-19

Robustness and Diversity Seeking Data-Free Knowledge Distillation

Artificial Intelligence Clinicians Can Use Chest Computed Tomography Technology to Automatically Diagnose Coronavirus Disease 2019 (COVID-19) Pneumonia and Enhance Low-Quality Images

Contact Info

Product

Resources

About