Data-Free Network Quantization With Adversarial Knowledge Distillation

Choi, Yoojin; Choi, Jihwan P.; El-Khamy, Mostafa; Lee, Jung Won

doi:10.1109/cvprw50498.2020.00363

Cited by 93 publications

(90 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Unified Framework Combining the above mentioned techniques will lead to a unified inversion framework [Choi et al, 2020] for data-free knowledge distillation:…”

Section: Preliminarymentioning

confidence: 99%

“…T. and S. refers to the scratch accuracy of teachers and students on the original training data. We compare our approach with the following baselines: DAFL , ZSKT [Micaelli and Storkey, 2019], ADI [Yin et al, 2020], DFQ [Choi et al, 2020] and LS-GDFD [Luo et al, 2020]. They all follow the unified framework discussed in Sec.…”

Section: Benchmarks On Knowledge Distillationmentioning

confidence: 99%

See 1 more Smart Citation

Contrastive Model Invertion for Data-Free Knolwedge Distillation

Fang

Song

Wang

et al. 2021

Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence

View full text Add to dashboard Cite

Model inversion, whose goal is to recover training data from a pre-trained model, has been recently proved feasible. However, existing inversion methods usually suffer from the mode collapse problem, where the synthesized instances are highly similar to each other and thus show limited effectiveness for downstream tasks, such as knowledge distillation. In this paper, we propose Contrastive Model Inversion (CMI), where the data diversity is explicitly modeled as an optimizable objective, to alleviate the mode collapse issue. Our main observation is that, under the constraint of the same amount of data, higher data diversity usually indicates stronger instance discrimination. To this end, we introduce in CMI a contrastive learning objective that encourages the synthesizing instances to be distinguishable from the already synthesized ones in previous batches. Experiments of pre-trained models on CIFAR-10, CIFAR-100, and Tiny-ImageNet demonstrate that CMI not only generates more visually plausible instances than the state of the arts, but also achieves significantly superior performance when the generated data are used for knowledge distillation. Code is available at https://github.com/zju-vipa/DataFree.

show abstract

“…Unified Framework Combining the above mentioned techniques will lead to a unified inversion framework [Choi et al, 2020] for data-free knowledge distillation:…”

Section: Preliminarymentioning

confidence: 99%

Section: Benchmarks On Knowledge Distillationmentioning

confidence: 99%

Contrastive Model Invertion for Data-Free Knolwedge Distillation

Fang

Song

Wang

et al. 2021

Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence

View full text Add to dashboard Cite

show abstract

“…We show the results for data-free PTQ (DF-PTQ) and data-free QAT (DF-QAT), both with ZS-CGAN. We compare our method with the existing DF-Q schemes in [11][12][13][14], and bold numbers indicate the best results among DF-Q schemes. For data-dependent quantization (DD-Q), we show the results for PTQ and QAT by using the original ImageNet dataset [8].…”

Section: Methodsmentioning

confidence: 99%

Zero-Shot Learning Of A Conditional Generative Adversarial Network For Data-Free Network Quantization

Choi

El-Khamy

Lee

2021

2021 IEEE International Conference on Image Processing (ICIP)

Self Cite

View full text Add to dashboard Cite

We propose a novel method for training a conditional generative adversarial network (CGAN) without the use of training data, called zero-shot learning of a CGAN (ZS-CGAN). Zero-shot learning of a conditional generator only needs a pre-trained discriminative (classification) model and does not need any training data. In particular, the conditional generator is trained to produce labeled synthetic samples whose characteristics mimic the original training data by using the statistics stored in the batch normalization layers of the pretrained model. We show the usefulness of ZS-CGAN in datafree quantization of deep neural networks. We achieved the state-of-the-art data-free network quantization of the ResNet and MobileNet classification models trained on the ImageNet dataset. Data-free quantization using ZS-CGAN showed a minimal loss in accuracy compared to that obtained by conventional data-dependent quantization.

show abstract

“…Secondly, additional re-training epochs are often needed in order to recover from the accuracy drop coused by quantization. To notice that re-training is time-consuming and the original training data are not always available for privacy or security reasons [31,32]. Recent trends suggest the use of alternative floating-point formats rather than integers, which require no extra fine-tuning steps, such as Bfloats [10] or reduced precision floating-point [33,34].…”

Section: Convnets Approximation Via Arithmetic Approximation and Data-reusementioning

confidence: 99%

AxP: A HW-SW Co-Design Pipeline for Energy-Efficient Approximated ConvNets via Associative Matching

Mocerino

Calimera

2021

Applied Sciences

View full text Add to dashboard Cite

The reduction in energy consumption is key for deep neural networks (DNNs) to ensure usability and reliability, whether they are deployed on low-power end-nodes with limited resources or high-performance platforms that serve large pools of users. Leveraging the over-parametrization shown by many DNN models, convolutional neural networks (ConvNets) in particular, energy efficiency can be improved substantially preserving the model accuracy. The solution proposed in this work exploits the intrinsic redundancy of ConvNets to maximize the reuse of partial arithmetic results during the inference stages. Specifically, the weight-set of a given ConvNet is discretized through a clustering procedure such that the largest possible number of inner multiplications fall into predefined bins; this allows an off-line computation of the most frequent results, which in turn can be stored locally and retrieved when needed during the forward pass. Such a reuse mechanism leads to remarkable energy savings with the aid of a custom processing element (PE) that integrates an associative memory with a standard floating-point unit (FPU). Moreover, the adoption of an approximate associative rule based on a partial bit-match increases the hit rate over the pre-computed results, maximizing the energy reduction even further. Results collected on a set of ConvNets trained for computer vision and speech processing tasks reveal that the proposed associative-based hw-sw co-design achieves up to 77% in energy savings with less than 1% in accuracy loss.

show abstract

Data-Free Network Quantization With Adversarial Knowledge Distillation

Cited by 93 publications

References 18 publications

Contrastive Model Invertion for Data-Free Knolwedge Distillation

Contrastive Model Invertion for Data-Free Knolwedge Distillation

Zero-Shot Learning Of A Conditional Generative Adversarial Network For Data-Free Network Quantization

AxP: A HW-SW Co-Design Pipeline for Energy-Efficient Approximated ConvNets via Associative Matching

Contact Info

Product

Resources

About