Robust Quantization: One Model to Rule Them All

Shkolnik, Moran; Chmiel, Brian; Banner, Ron; Shomron, Gil; Nahshan, Yury; Bronstein, Alex; Weiser, Uri

doi:10.48550/arxiv.2002.07686

Cited by 3 publications

(5 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This fact, that in theory is not guaranteed, in practice is well established especially for the models trained from scratch, where the initial parameters distributions are typically Normal-like and surely well verified for the models considered in the Intel-Habana Labs paper, which are mostly based on convolutional architectures. Their paper [1] concludes with the statement:…”

Section: A Very Smart Idea From Our Colleagues In Intel-habana Labsmentioning

confidence: 94%

“…Intel-Habana Labs have recently published a paper [1] which considers the problem of steering the training procedure in such a way that the training itself is guided to finding solutions (that is, a set of parameters in parameter space) such that the subsequent quantization of the model is minimally harmful. Their paper considers the case of uniform symmetrical quantization and shows that in such settings, among all possible distributions, the uniform distribution is the one that yields the minimal quantization sensitivity (theorem 4 in the paper).…”

Section: A Very Smart Idea From Our Colleagues In Intel-habana Labsmentioning

confidence: 99%

“…Here we present a case of application of Kurtosis as regularizer that is not considered in the reference paper [1], but which is of practical interest.…”

Section: A Very Smart Idea From Our Colleagues In Intel-habana Labsmentioning

confidence: 99%

See 2 more Smart Citations

Quantization Aware Training, ERNIE and Kurtosis Regularizer: a short empirical study

Zanetti

2021

Preprint

View full text Add to dashboard Cite

Pre-trained language models like ERNIE or BERT [2] are currently used in many applications. These models come with a set of pre-trained weights typically obtained in unsupervised/self-supervised modality on a huge amount of data. After that, they are fine-tuned on a specific task. Applications then use these models for inference, and often some additional constraints apply, like low power-budget or low latency between input and output. The main avenue to meet these additional requirements for the inference settings, is to use low precision computation (e.g. INT8 rather than FP32), but this comes with the cost of deteriorating the functional performance (e.g. accuracy) of the model. Some approaches have been developed to tackle the problem and go beyond the limitations of the PTO (Post-Training Quantization), more specifically the QAT (Quantization Aware Training, see [4]) is a procedure that "interferes" with the training process in order to make it affected (or simply 'disturbed') by the quantization phase during the training itself. Besides QAT, recently Intel-Habana Labs have proposed an additional and more direct way to make the training results more robust to subsequent quantization which uses a regularizer, therefore changing the loss function that drives the training procedure. But their proposal does not work "out of the box" for pre-trained models like ERNIE, for example. In this short paper we show why this is not happening (for the ERNIE case) and we propose a very basic way to deal with it, sharing as well some initial results (increase in final INT8 accuracy) that might be of interest to practitioners willing to use ERNIE in their applications, in low precision regime.

show abstract

Section: A Very Smart Idea From Our Colleagues In Intel-habana Labsmentioning

confidence: 94%

Section: A Very Smart Idea From Our Colleagues In Intel-habana Labsmentioning

confidence: 99%

See 1 more Smart Citation

Quantization Aware Training, ERNIE and Kurtosis Regularizer: a short empirical study

Zanetti

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Later, [43] finds that quantized DNNs are actually more vulnerable to adversarial attacks due to the error amplification effect, i.e., the magnitude of adversarial perturbation is amplified when passing through the DNN layers. To tackle this effect, [43,68] propose robustness-aware regularization methods for DNN training, and [69] retrains the network via feedback learning [70]. In addition, [55] searches for layerwise precision and [26] constructs a unified formulation to balance and enforce the models' robustness and compactness.…”

Section: Related Workmentioning

confidence: 99%

2-in-1 Accelerator: Enabling Random Precision Switch for Winning Both Adversarial Robustness and Efficiency

Zhao

et al. 2021

MICRO-54: 54th Annual IEEE/ACM International Symposium on Microarchitecture

View full text Add to dashboard Cite

The recent breakthroughs of deep neural networks (DNNs) and the advent of billions of Internet of Things (IoT) devices have excited an explosive demand for intelligent IoT devices equipped with domainspecific DNN accelerators. However, the deployment of DNN accelerator enabled intelligent functionality into real-world IoT devices still remains particularly challenging. First, powerful DNNs often come at prohibitive complexities, whereas IoT devices often suffer from stringent resource constraints. Second, while DNNs are vulnerable to adversarial attacks especially on IoT devices exposed to complex real-world environments, many IoT applications require strict security. Existing DNN accelerators mostly tackle only one of the two aforementioned challenges (i.e., efficiency or adversarial robustness) while neglecting or even sacrificing the other. To this end, we propose a 2-in-1 Accelerator, an integrated algorithm-accelerator co-design framework aiming at winning both the adversarial robustness and efficiency of DNN accelerators. Specifically, we first propose a Random Precision Switch (RPS) algorithm that can effectively defend DNNs against adversarial attacks by enabling random DNN quantization as an in-situ model switch during training and inference. Furthermore, we propose a new precision-scalable accelerator featuring (1) a new precision-scalable MAC unit architecture which spatially tiles the temporal MAC units to boost both the achievable efficiency and flexibility and (2) a systematically optimized dataflow that is searched by our generic accelerator optimizer. Extensive experiments and ablation studies validate that our 2-in-1 Accelerator can not only aggressively boost both the adversarial robustness and efficiency of DNN accelerators under various attacks, but also naturally support instantaneous robustness-efficiency trade-offs adapting to varied resources without the necessity of DNN retraining. We believe our 2-in-1 Accelerator has opened up an exciting perspective for robust and efficient accelerator design. CCS CONCEPTS• Computer systems organization → Neural networks.

show abstract

“…NB-SMT mitigates hardware underutiliation of DNNs caused by unstructured sparsity. In contrast to GPPs, which are obliged to keep program semantics so as to produce consistent results, DNNs are able to cope with, for example, pruning of activations and weights [5,20] and with quantization [1,17] without retraining. NB-SMT exploits DNN resiliency to avoid backpressure [19] by "squeezing" more than one thread into the shared resource by temporarily reducing the threads' numerical precision.…”

Section: Introductionmentioning

confidence: 99%

Post-Training BatchNorm Recalibration

Shomron,

Weiser

2020

Preprint

Self Cite

View full text Add to dashboard Cite

We revisit non-blocking simultaneous multithreading (NB-SMT) introduced previously by Shomron and Weiser [18]. NB-SMT trades accuracy for performance by occasionally "squeezing" more than one thread into a shared multiply-andaccumulate (MAC) unit. However, the method of accommodating more than one thread in a shared MAC unit may contribute noise to the computations, thereby changing the internal statistics of the model. We show that substantial model performance can be recouped by post-training recalibration of the batch normalization layers' running mean and running variance statistics, given the presence of NB-SMT.

show abstract

Robust Quantization: One Model to Rule Them All

Cited by 3 publications

References 0 publications

Quantization Aware Training, ERNIE and Kurtosis Regularizer: a short empirical study

Quantization Aware Training, ERNIE and Kurtosis Regularizer: a short empirical study

2-in-1 Accelerator: Enabling Random Precision Switch for Winning Both Adversarial Robustness and Efficiency

Post-Training BatchNorm Recalibration

Contact Info

Product

Resources

About