SinReQ: Generalized Sinusoidal Regularization for Low-Bitwidth Deep Quantized Training

Abstract

Deep quantization of neural networks (below eight bits) offers significant promise in reducing their compute and storage cost. Albeit alluring, without special techniques for training and optimization, deep quantization results in significant accuracy loss. To further mitigate this loss, we propose a novel sinusoidal regularization, called SinReQ, for deep quantized training. SinReQ adds a periodic term to the original objective function of the underlying training algorithm. SinReQ exploits the periodicity, differentiability, and the desired convexity profile in sinusoidal functions to automatically propel weights towards values that are inherently closer to quantization levels. Since, this technique does not require invasive changes to the training procedure, SinReQ can harmoniously enhance quantized training algorithms. SinReQ offers generality and flexibility as it is not limited to a certain bitwidth or a uniform assignment of bitwidths across layers. We carry out experimentation using the CIFAR-10, ResNet-20, SVHN DNNs with three to five bits for quantization and show the versatility of SinReQ in enhancing multiple quantized training algorithms, DoReFa (Zhou et al., 2016) and WRPN (Mishra et al., 2018). Averaging across all the bit configurations shows that SinReQ closes the accuracy gap between these two techniques and the full-precision runs by 35.7% and 37.1%, respectively. That is improving the absolute accuracy of DoReFa and WRPN up to 5.3% and 2.6%, respectively.

Quantization-Guided Training for Compact TinyML Models

“…Such an approach, however, does not lend itself well to non-binary representations and also non-linear quantization schemes. Elthakeb et al [9] alleviate the former issue by using a sinusoidal regularizer on the quantized weights.…”

“…There is no dependence on the use of STEs, which greatly improves ease of implementation. Also, compared to previous similar regularizer-based approaches [9,12,33], since in QGT the regularizer is applied on the weight values directly rather than the quantized values, there is no need to learn the scale of the quantized weights separately. Using regularizers, QGT can enforce properties such as clustering of weight values into quantized bins, which can accommodate non-linear, hardwarespecific quantizers.…”

“…Regularizers have been used in DNN training [1,13], even to target DNNs with binary-value weights [12,33]. The main novelties of our QGT approach are that, unlike these previous approaches that target the quantized weights [9], we focus on the dequantized weights, which circumvents the need to learn scale and intercept separately. Further, our approach can accommodate custom nonlinear quantizations appropriate for custom hardware.…”