Post-Training Sparsity-Aware Quantization

Abstract

Quantization is a technique used in deep neural networks (DNNs) to increase execution performance and hardware efficiency. Uniform post-training quantization (PTQ) methods are common, since they can be implemented efficiently in hardware and do not require extensive hardware resources or a training set. Mapping FP32 models to INT8 using uniform PTQ yields models with negligible accuracy degradation; however, reducing precision below 8 bits with PTQ is challenging, as accuracy degradation becomes noticeable, due to the increase in quantization noise. In this paper, we propose a sparsity-aware quantization (SPARQ) method, in which the unstructured and dynamic activation sparsity is leveraged in different representation granularities. 4-bit quantization, for example, is employed by dynamically examining the bits of 8-bit values and choosing a window of 4 bits, while first skipping zero-value bits. Moreover, instead of quantizing activation-by-activation to 4 bits, we focus on pairs of 8-bit activations and examine whether one of the two is equal to zero. If one is equal to zero, the second can opportunistically use the other's 4-bit budget; if both do not equal zero, then each is dynamically quantized to 4 bits, as described. SPARQ achieves minor accuracy degradation, 2× speedup over widely used hardware architectures, and a practical hardware implementation. The code is available at https://github.com/gilshm/sparq.Preprint. Under review.

Towards Real-Time Machine Learning-Based Signal/Background Selection in the CMS Detector Using Quantized Neural Networks and Input Data Reduction

“…According to [54], the usage of low-precision fixed integer value representation has the potential to reduce the memory footprint and latency by a factor of 16×. Despite being fast and very easy to use, the post-training quantization approach is not an option because it suffers from significant degradation in model accuracy in case of precision lower than 8 bits [55].…”

Characterizing and Understanding the Behavior of Quantized Models for Reliable Deployment

“…Model quantization is an optimization technique that aims at transforming the higher-bit level weights to lower-bit level weights, e.g., from float32 weights to 8-bit integer weights, to reduce the size of the model for an easy model deployment. Multiple quantization approaches [32,33,38,53] have been proposed given its importance in DL-based engineering. An important part of quantization methods is the mapping between the two parts of weights.…”

An Efficient CNN Inference Accelerator Based on Intra- and Inter-Channel Feature Map Compression

“…To reduce the data exchange bandwidth and on-chip storage size requirements for CNN inference accelerators, one straightforward method is to compress the generated interlayer data simultaneously during inference. Many works reported quantizing the activation into low-precision fixed-point data to reduce the interlayer data size [11,25,26,54,55]. However, the lowprecision activation will significantly degrade the prediction accuracy of CNNs if the quantization is straightforwardly implemented on interlayer activations without retraining [27,28].…”