RTN: Reparameterized Ternary Network

Abstract

To deploy deep neural networks on resource-limited devices, quantization has been widely explored. In this work, we study the extremely low-bit networks which have tremendous speed-up, memory saving with quantized activation and weights. We first bring up three omitted issues in extremely low-bit networks: the squashing range of quantized values; the gradient vanishing during backpropagation and the unexploited hardware acceleration of ternary networks. By reparameterizing quantized activation and weights vector with full precision scale and offset for fixed ternary vector, we decouple the range and magnitude from direction to extenuate above problems. Learnable scale and offset can automatically adjust the range of quantized values and sparsity without gradient vanishing. A novel encoding and computation pattern are designed to support efficient computing for our reparameterized ternary network (RTN). Experiments on ResNet-18 for ImageNet demonstrate that the proposed RTN finds a much better efficiency between bitwidth and accuracy and achieves up to 26.76% relative accuracy improvement compared with state-of-the-art methods. Moreover, we validate the proposed computation pattern on Field Programmable Gate Arrays (FPGA), and it brings 46.46 × and 89.17 × savings on power and area compared with the full precision convolution.

FAT: An In-Memory Accelerator with Fast Addition for Ternary Weight Neural Networks

“…The weights are ternarized to {+1, 0, -1} by comparing with trained or given thresholds as equation ( 4) shows, where w and w t are the original and ternarized weight values, and T H low and T H high are the thresholds. Modern TWNs train the weights to be ternary values [14], [15], [16], so the weights of target neural networks are already quantized into 2-bit numbers when the training finishes.…”

“…BWNs quantize the weights of CNNs into {+1, -1} to replace the computation-intensive multiplication operations with addition and subtraction operations for high speedup, but this aggressive quantization also leads to lower accuracy. As both the accuracy and the speed of CNNs matter, other quantization methods, including 8-bit [10], [11] and 4-bit [12], [13] integer quantization (INT8 and INT4) and ternary quantization [14], [15], [16], are proposed to do a trade-off between the speed and accuracy as Table . I shows.…”

“…Ternary Weight Networks (TWNs) [14], [15] make an excellent trade-off between BWNs and 32-bit Full-Precision (FP) CNNs. TWNs quantize the weights of CNNs into {+1, 0, -1} and brings several advantages, as shown in Table . I.…”

Binary neural networks for image super-resolution

“…XNOR-Net++ [27] 则将二值卷积过程中的权值与激活的尺度因子融合为一个可通过网络自适应学习的参数, 解决 XNOR 中由于固定尺度因子造成的激活值被限制在固定区间的问题. 与 XNOR-Net++ 方法类似, RTN [28] 网 络提出了重新参数化量化数值的总体流程, 量化后的激活由一个尺度因子重新调整大小, 同时由一个 偏移因子重新调整区间, 以实现更高的网络容量. 此外, 对于 1-bit 网络而言, 性能对于激活值的特征 分布十分敏感.…”

Few Shot Network Compression via Cross Distillation

“…Deep neural networks (DNNs) have achieved remarkable success in a wide range of applications, however, they suffer from substantial computation and energy cost. In order to obtain light-weighted DNNs, network compression techniques have been widely developed in recent years, including network pruning (He, Zhang, and Sun 2017;Luo, Wu, and Lin 2017;Wen et al 2019), quantization (Han, Mao, and Dally 2016;Wu et al 2016;Li et al 2020) and knowledge distillation (Hinton, Vinyals, and Dean 2015;Romero et al 2014).…”