Fighting Quantization Bias With Bias

Abstract

Low-precision representation of deep neural networks (DNNs) is critical for efficient deployment of deep learning application on embedded platforms, however, converting the network to low precision degrades its performance. Crucially, networks that are designed for embedded applications usually suffer from increased degradation since they have less redundancy. This is most evident for the ubiquitous MobileNet architecture [10,20] which requires a costly quantization-aware training cycle to achieve acceptable performance when quantized to 8-bits. In this paper, we trace the source of the degradation in MobileNets to a shift in the mean activation value. This shift is caused by an inherent bias in the quantization process which builds up across layers, shifting all network statistics away from the learned distribution. We show that this phenomenon happens in other architectures as well. We propose a simple remedy -compensating for the quantization induced shift by adding a constant to the additive bias term of each channel. We develop two simple methods for estimating the correction constants -one using iterative evaluation of the quantized network and one where the constants are set using a short training phase. Both methods are fast and require only a small amount of unlabeled data, making them appealing for rapid deployment of neural networks. Using the above methods we are able to match the performance of trainingbased quantization of MobileNets at a fraction of the cost.

HPTQ: Hardware-Friendly Post Training Quantization

“…Bias Correction. Quantization of weights induce bias shifts to activation means that may lead to detrimental behaviour in the following layers [23,32]. can be expressed as follows:…”

“…Several works propose approaches to correct the quantization induced bias. These include using batch-normalization statistics [23], micro training [32] and applying scale and shift per channel [33].…”

Q-Rater: Non-Convex Optimization for Post-Training Uniform Quantization

“…Unfortunately, because the task loss function of DNNs is non-convex (Goodfellow et al, 2016), there is no analytical solution to find the form of post-training quantized weights. As a result, various approximations are being suggested mainly by using quadratic approximations that imply quantization is performed in a convex-like regime (Nagel et al, 2017;Nahshan et al, 2020).…”

“…Bias correction is an operation to compensate for the biased error in output activations after quantization. The amount of shift induced by quantization is diminished by adjusting the bias parameters of the neurons or channels because shifted output activations through quantization may degrade the quantization quality of the next layer (Finkelstein et al, 2019;Nagel et al, 2019). The amount of shift can be calculated as the expected error on the output activations that can be expressed as γ n = -0.9…”

“…Bias correction has been a supplementary and optional technique for quantization. For example, bias correction is not introduced in (Zhao et al, 2019) while it is playing a key role in enhancing model accuracy in (Finkelstein et al, 2019;Nagel et al, 2019). In the context of non-convexity, Q-Rater compares two model accuracy values evaluated with or without bias correction for a layer.…”