Least squares binary quantization of neural networks

Pouransari, Hadi; Tu, Zhucheng; Tuzel, Oncel

doi:10.1109/cvprw50498.2020.00357

Cited by 24 publications

(18 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Quantization schemes. To show SumMerge robustness, we evaluate performance on six quantization schemes (with different values), across several DNN architectures and datasets [8,12,32,33,39,42]-shown in Table 1. For each entry in the table, we used code from each work's public repository to train and quantize each DNN.…”

Section: Methodsmentioning

confidence: 99%

“…While each scheme quantizes to a different set of weights, the primary concern for Sum-Merge is the effective value, the resulting weight distribution (Section 3.6) and whether one of the weights is 0 (Section 3.5). For example, TTQ [42] and LS-T [32] quantize to = 3 weights: { , , 0} and {− , , 0}, respectively, where and are positive and negative per-layer weights and is a per-filter weight (Section 3.5). The difference in values between , and does not impact performance, but the fact that each scheme includes the 0-valued weight, and the weight distribution of each layer, does impact performance.…”

Section: Methodsmentioning

confidence: 99%

“…There has been much recent work on quantization schemes that reduce the number of bits needed to represent the weights (and activations) of DNNs while preserving model accuracy [8,12,13,18,32,33,37,[39][40][41][42]. This decreases the memory footprint of DNNs and can also be exploited to increase ALU throughput (e.g., for SIMD implementations, we can fit more elements in a fixed bit-width vector).…”

Section: Quantization and Activation Group Reusementioning

confidence: 99%

“…Per-layer vs. per-filter weights. Depending on the quantization scheme, the same weights may be shared across all filters (e.g., TTQ [42]) or each filter may have a distinct weights (e.g., LQ [39] and LS [32]). Up to this point, the description has been consistent with the former case.…”

Section: Other Considerationsmentioning

confidence: 99%

See 3 more Smart Citations

SumMerge

Prabhakar

Kuhar

Agrawal

et al. 2021

Proceedings of the ACM International Conference on Supercomputing

View full text Add to dashboard Cite

Deep Neural Network (DNN) inference efficiency is a key concern across the myriad of domains now relying on Deep Learning. A recent promising direction to speed-up inference is to exploit weight repetition. The key observation is that due to DNN quantization schemes-which attempt to reduce DNN storage requirements by reducing the number of bits needed to represent each weight-the same weight is bound to repeat many times within and across filters. This enables a weight-repetition aware inference kernel to factorize and memoize out common sub-computations, reducing arithmetic per inference while still maintaining the compression benefits of quantization. Yet, significant challenges remain. For instance, weight repetition introduces significant irregularity in the inference operation and hence (up to this point) has required custom hardware accelerators to derive net benefit. This paper proposes SumMerge: a new algorithm and set of implementation techniques to make weight repetition practical on general-purpose devices such as CPUs. The key idea is to formulate inference as traversing a sequence of data-flow graphs with weight-dependent structure. We develop an offline heuristic to select a data-flow graph structure that minimizes arithmetic operations per inference (given trained weight values) and use an efficient online procedure to traverse each data-flow graph and compute the inference result given DNN inputs. We implement the above as an optimized C++ routine that runs on a commercial multicore processor with vector extensions and evaluate performance relative to Intel's optimized library oneDNN and the prior-art weight repetition algorithm (AGR). When applied on top of six different quantization schemes, SumMerge achieves a speedup of between

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Methodsmentioning

confidence: 99%

Section: Quantization and Activation Group Reusementioning

confidence: 99%

Section: Other Considerationsmentioning

confidence: 99%

See 2 more Smart Citations

SumMerge

Prabhakar

Kuhar

Agrawal

et al. 2021

Proceedings of the ACM International Conference on Supercomputing

View full text Add to dashboard Cite

show abstract

“…As Table . 1 shows, ResNet-18 quantized as TNN and TBN have 7.6% and 3.9% higher absolute Top-1 accuracy on ImageNet than binarized ResNet-18 respectively. [33] Ternary/Binary 32× >16× 62.0% 83.6% BNN CI-BCNN [44] Binary (1-bit) 32× 64× 58.1% 80.0%…”

Section: Introductionmentioning

confidence: 99%

TAB: Unified and Optimized Ternary, Binary, and Mixed-precision Neural Network Inference on the Edge

Zhu

Duong

Liu

2022

ACM Trans. Embed. Comput. Syst.

View full text Add to dashboard Cite

Ternary Neural Networks (TNNs) and mixed-precision Ternary Binary Networks (TBNs) have demonstrated higher accuracy compared to Binary Neural Networks (BNNs) while providing fast, low-power and memory-efficient inference. Related works have improved the accuracy of TNNs and TBNs, but overlooked their optimizations on CPU and GPU platforms. First, there is no unified encoding for the binary and ternary values in TNNs and TBNs. Second, existing works store the 2-bit quantized data sequentially in 32/64-bit integers, resulting in bit-extraction overhead. Last, adopting standard 2-bit multiplications for ternary values leads to a complex computation pipeline, and efficient mixed-precision multiplication between ternary and binary values is unavailable. In this paper, we propose TAB as a unified and optimized inference method for ternary, binary and mixed-precision neural networks. TAB includes unified value representation, efficient data storage scheme, and novel bitwise dot product pipelines on CPU/GPU platforms. We adopt signed integers for consistent value representation across binary and ternary values. We introduce a bitwidth-last data format that stores the first and second bits of the ternary values separately to remove the bit extraction overhead. We design the ternary and binary bitwise dot product pipelines based on Gated-XOR using up to 40% fewer operations than State-Of-The-Art (SOTA) methods. Theoretical speedup analysis shows that our proposed TAB-TNN is 2.3 × fast as the SOTA ternary method RTN, 9.8 × fast as 8-bit integer quantization (INT8), and 39.4 × fast as 32-bit full-precision convolution (FP32). Experiment results on CPU and GPU platforms show that our TAB-TNN has achieved up to 34.6 × speedup and 16 × storage size reduction compared with FP32 layers. TBN, Binary-activation Ternary-weight Network (BTN) and BNN in TAB are up to 40.7 ×, 56.2 × and 72.2 × fast as FP32. TAB-TNN is up to 70.1% faster and 12.8% more power-efficient than RTN on Darknet-19 while keeping the same accuracy. TAB is open source as a PyTorch Extension 1 for easy integration with existing CNN models.

show abstract