Same, Same But Different - Recovering Neural Network Quantization Error Through Weight Factorization

Abstract

Quantization of neural networks has become common practice, driven by the need for efficient implementations of deep neural networks on embedded devices. In this paper, we exploit an oft-overlooked degree of freedom in most networks -for a given layer, individual output channels can be scaled by any factor provided that the corresponding weights of the next layer are inversely scaled. Therefore, a given network has many factorizations which change the weights of the network without changing its function. We present a conceptually simple and easy to implement method that uses this property and show that proper factorizations significantly decrease the degradation caused by quantization. We show improvement on a wide variety of networks and achieve state-of-the-art degradation results for MobileNets. While our focus is on quantization, this type of factorization is applicable to other domains such as network-pruning, neural nets regularization and network interpretability.

HPTQ: Hardware-Friendly Post Training Quantization

“…Activation Equalization. In this step, we equalize activation ranges per channel similarly to the methods presented in [23,28]. Here, we set the scale-perchannel factor according to the value of the threshold that is selected per-tensor.…”

“…The motivation to use this scaling factor in order to equalize the activation ranges is to use the maximum range of the quantization bins for each channel (see Figure 4). The authors in [23,28] suggest to perform channel equalization by exploiting the positive scale equivariance property of activation functions. It holds for any piece-wise linear activation function in its relaxed form: φ (Sx) = S φ (x) where φ is a piece-wise linear function, φ is its modified version that fits this requirement and S = diag (s) is a diagonal matrix with s k denoting the scale factor for channel k.…”

“…Method PC PoT F-Acc Q-Acc ∆ QAT QT [10] 70.9 70.0 0.9 TQT [11] 71.1 71.1 0.0 PTQ SSBD [28] 70.9 69.95 0.95 Krishnamoorthi [30] 70.9 70.3 0.6 Wu et al [35] 71.88 70.39 1.49 Lee et al [36] 69.5 68.84 0.66 HPTQ (Our) 70.55 70.41 0.14 Type Method PC PoT F-Acc Q-Acc ∆ QAT QT [10] 71.9 70.9 1.0 RVQuant [37] 70.10 70.29 -0.19 TQT [11] 71.7 71.8 -0.10 PTQ AdaQuant [38] 73.03 73.03 0.0 ZeroQ [15] 73.03 72.91 0.12 SSBD [28] 71.9 71.29 0.61 Wu et al [35] 71.88 71.14 0.74 Krishnamoorthi [30] 71…”

“…QT [10] 76.4 74.9 1.5 RVQuant [37] 75.92 75.67 0.25 HAWQ-V3 [39] 77.72 77.58 0.14 LSQ [40] 76.9 76.8 0.1 TQT [11] 76.9 76.5 0.4 FAQ [41] 75.4 75.4 0.0 PTQ ZeroQ [15] 77.72 77.67 0.05 OCS [42] 76.1 75.9 0.2 SSBD [28] 75.2 74.95 0.25 He et al [43] 75.3 75.03 0.27 Wu et al [35] 76 Object Detection. We evaluate HPTQ on COCO [45] using the SSD detector [4] with several backbones 3 .…”

Feature Map Transform Coding for Energy-Efficient CNN Inference

“…Banner et al (2018) derived an analytical expression for the approximation of the optimal threshold under the assumption of Laplacian or Gaussian distribution of the weights, which allows to achieve single percent accuracy reduction for 8-bit weights and 4-bit activations. Meller et al (2019) showed that the equalization of channels and the removal of outliers allowed to improve quantization quality. Choukroun et al (2019) used one-dimensional exact line-search to evaluate optimal quantization threshold, demonstrating state-ofthe-art results for 4-bit weight and activation quantization.…”