A White Paper on Neural Network Quantization

Abstract

While neural networks have advanced the frontiers in many applications, they often come at a high computational cost. Reducing the power and latency of neural network inference is key if we want to integrate modern networks into edge devices with strict power and compute requirements. Neural network quantization is one of the most effective ways of achieving these savings but the additional noise it induces can lead to accuracy degradation. In this white paper, we introduce state-of-the-art algorithms for mitigating the impact of quantization noise on the network's performance while maintaining low-bit weights and activations. We start with a hardware motivated introduction to quantization and then consider two main classes of algorithms: Post-Training Quantization (PTQ) and Quantization-Aware-Training (QAT). PTQ requires no re-training or labelled data and is thus a lightweight push-button approach to quantization. In most cases, PTQ is sufficient for achieving 8-bit quantization with close to floating-point accuracy. QAT requires fine-tuning and access to labeled training data but enables lower bit quantization with competitive results. For both solutions, we provide tested pipelines based on existing literature and extensive experimentation that lead to state-of-the-art performance for common deep learning models and tasks.

Post-Training Quantization for Cross-Platform Learned Image Compression

“…When v is the activation output, the second term can be pre-computed and absorbed by the convolution bias, so it will not introduce extra calculation to inference. Therefore, it is recommended by Nagel et al (2021) to apply asymmetric quantization on activation and symmetric quantization on weights. Nagel et al (2021) also suggests to adopt a per-channel quantization on weights (Krishnamoorthi, 2018; Li et al, 2019).…”

“…Therefore, it is recommended by Nagel et al (2021) to apply asymmetric quantization on activation and symmetric quantization on weights. Nagel et al (2021) also suggests to adopt a per-channel quantization on weights (Krishnamoorthi, 2018; Li et al, 2019).…”

“…This benefits the CDF indexing and calculation which we will discuss later in Section 4.1. As recommended in Nagel et al (2021), we adopt a grid search minimizing the reconstruction error to obtain the weight quantization step s W of each layer. We figure out the activation quantization step s u with Min-Max method on a little calibration data.…”

“…When adopting our approach or on model, the RD performance marginally deteriorates. However, adopts a per-channel activation quantization which is unfriendly to hardware implementation and rarely supported(Nagel et al, 2021). When adopting with per-tensor activation quantization, the performance, especially at higher bit-rates, gets hurt significantly.…”

F8Net: Fixed-Point 8-bit Only Multiplication for Network Quantization

“…Real-time inference on resource-constrained and efficiency-demanding platforms has long been desired and extensively studied in the last decades, resulting in significant improvement on the trade-off between efficiency and accuracy (Han et al, 2015;Mei et al, 2019;Tanaka et al, 2020;Ma et al, 2020;Mishra et al, 2020;Liang et al, 2021;Liu et al, 2021). As a model compression technique, quantization is promising compared to other methods, such as network pruning (Tanaka et al, 2020;Ma et al, 2020; and slimming (Liu et al, 2017;2018), as it achieves a large compression ratio (Krishnamoorthi, 2018;Nagel et al, 2021) and is computationally beneficial for integer-only hardware. The latter one is especially important because many hardwares (e.g., most brands of DSPs (Ho, 2015;QCOM, 2019)) only support integer or fixed-point arithmetic for accelerated implementation and cannot deploy models with floating-point operations.…”