In-Hindsight Quantization Range Estimation for Quantized Training

Abstract

Quantization techniques applied to the inference of deep neural networks have enabled fast and efficient execution on resource-constraint devices. The success of quantization during inference has motivated the academic community to explore fully quantized training, i.e. quantizing backpropagation as well. However, effective gradient quantization is still an open problem. Gradients are unbounded and their distribution changes significantly during training, which leads to the need for dynamic quantization. As we show, dynamic quantization can lead to significant memory overhead and additional data traffic slowing down training. We propose a simple alternative to dynamic quantization, in-hindsight range estimation, that uses the quantization ranges estimated on previous iterations to quantize the present. Our approach enables fast static quantization of gradients and activations while requiring only minimal hardware support from the neural network accelerator to keep track of output statistics in an online fashion. It is intended as a drop-in replacement for estimating quantization ranges and can be used in conjunction with other advances in quantized training. We compare our method to existing methods for range estimation from the quantized training literature and demonstrate its effectiveness with a range of architectures, including MobileNetV2, on image classification benchmarks (Tiny ImageNet & ImageNet).

Logarithmic Unbiased Quantization: Simple 4-bit Training in Deep Learning

“…Also, Sun et al ( 2019) presented a novel hybrid format for full training in FP8, while the weights and activations are quantized to [1,4,3] format, the neural gradients are quantized to [1,5,2] format to catch a wider dynamic range. Fournarakis & Nagel (2021) suggested a method to reduce the data traffic during the calculation of the quantization range.…”

“…In addition, this issue might be solved with dedicated hardware such as a unit that calculates the statistics value more efficiently or use memory-on-chip blocks which reduce data-movement overhead. A recent method (Fournarakis & Nagel, 2021) tries to reduce the data movement by using previous iterations statistic, but as shown in Fig. 5a in the appendix, combining it with LUQ cause accuracy degradation.…”

“…This statistics measurement increase the data movement from and to memory, in a similar way than previous suggested methods (Sun et al, 2020). Recently, Fournarakis & Nagel (2021) suggests "in-hindsight" method to reduce the data movement overhead that occurs in the calculation on-the-fly of the quantization ranges by using a running average of the previous iterations statistics. In Fig.…”