Extremely Low Bit Transformer Quantization for On-Device Neural Machine Translation

Abstract

The deployment of widely used Transformer architecture is challenging because of heavy computation load and memory overhead during inference, especially when the target device is limited in computational resources such as mobile or edge devices. Quantization is an effective technique to address such challenges. Our analysis shows that for a given number of quantization bits, each block of Transformer contributes to translation quality and inference computations in different manners. Moreover, even inside an embedding block, each word presents vastly different contributions. Correspondingly, we propose a mixed precision quantization strategy to represent Transformer weights by an extremely low number of bits (e.g., under 3 bits). For example, for each word in an embedding block, we assign different quantization bits based on statistical property. Our quantized Transformer model achieves 11.8× smaller model size than the baseline model, with less than -0.5 BLEU. We achieve 8.3× reduction in run-time memory footprints and 3.5× speed up (Galaxy N10+) such that our proposed compression strategy enables efficient implementation for on-device NMT.

nuQmm: Quantized MatMul for Efficient Inference of Large-Scale Generative Language Models

“…For the Transformers and their variants, the processing time of matrix multiplications dominates the entire inference latency because of higher time complexity compared to activation functions, normalization layers, and so on [27], [28], [30]. To validate such a claim, Fig.…”

“…which does not have analytical solutions except when q = 1. Thus, scaling factors and binary vectors are obtained by iterative search methods [20], [34] or by quantization-aware training [30]. In this work, we discuss the unique mathematical properties of BCQ to enable efficient quantized matrix multiplications.…”

“…Again, an (m × m) matrix is multiplied by an (m × 1) matrix while we select m to be 8192, 12288 (used for GPT-3 175B), or 16384. For Table IV, we include the case of q = 2 for nuQmm (with g = m) as 2-bit quantization for the Transformer is reported to be feasible by quantization-aware training along with BCQ format [30]. We notice that throughout all m configurations, increasing GPUs for cuBLAS with tensor parallelism brings about a higher reduction in GPU utilization, memory utilization, and latency ratio of computations.…”

The Low-Resource Double Bind: An Empirical Study of Pruning for Low-Resource Machine Translation

“…quantization or faster search algorithms. Quantization can be used to speed up inference and relax hardware requirements, as has been shown for e.g., 8-bit (Quinn andBallesteros, 2018), 4-bit (Aji and and recently also below 3-bit quantization (Chung et al, 2020) of NMT models. In the wider NLP space, there has been interest in evaluating the trade-offs of different compression techniques for downstream finetuning.…”

Bag of Tricks for Optimizing Transformer Efficiency

“…Experiments on the WMT14 En-De, WMT14 En-Fr and NIST12 Zh-En machine translation (MT) benchmarks demonstrate that the improved system achieves a 3.80× speedup on CPU and a 2.52× speedup on GPU with performance on par with the baseline. The speedup obtained is available on most modern hardware, as it does not depend on specific hardware or library, e.g., quantization (Chung et al, 2020) and unstructured pruning (Hoefler et al, 2021) require the support of the latest hardware-dependent and acceleration libraries.…”