BinaryBERT: Pushing the Limit of BERT Quantization

Abstract

The rapid development of large pre-trained language models has greatly increased the demand for model compression techniques, among which quantization is a popular solution. In this paper, we propose BinaryBERT, which pushes BERT quantization to the limit with weight binarization. We find that a binary BERT is hard to be trained directly than a ternary counterpart due to its complex and irregular loss landscapes. Therefore, we propose ternary weight splitting, which initializes the binary model by equivalent splitting from a half-sized ternary network. The binary model thus inherits the good performance of the ternary model, and can be further enhanced by fine-tuning the new architecture after splitting. Empirical results show that BinaryBERT has negligible performance drop compared to the full-precision BERT-base while being 24× smaller, achieving the state-of-the-art results on GLUE and SQuAD benchmarks.

VAQF: Fully Automatic Software-Hardware Co-Design Framework for Low-Bit Vision Transformer

“…Since the BERT era (Vaswani et al, 2017), quantization has been extensively studied to reduce the memory and computation complexity of the transformer architectures (Prato et al, 2019;Zafrir et al, 2019;Shen et al, 2020;Zhang et al, 2020;Bai et al, 2020). In detail, (Zafrir et al, 2019) finetunes the BERT with 8-bit quantization-aware training, and successfully compresses BERT with minimal accuracy loss.…”

“…The later Ternary-BERT (Zhang et al, 2020) proposes to use approximation based and loss-aware ternarization to ternarize the weights in the BERT, and use distillation to further reduce the accuracy drop caused by lower capacity. The BinaryBERT (Bai et al, 2020) suggests that it is difficult to train a binary BERT directly due to its complex loss landscape, and proposes ternary weight splitting strategy to make the binary BERT inherit the good performance of the ternary one. However, all of them are designed for NLP, not for the computer vision tasks.…”

“…Similar to (Liu et al, 2020;Bai et al, 2020), we binarize the weights in ViT, and reduce the activations into low-precision, to achieve the trade-off between efficiency and accuracy.…”

“…To address this issue, many efforts have been devoted to compressing the cumbersome transformer architectures into a lightweight counterpart, including knowledge distillation (Jiao et al, 2019;Sanh et al, 2019;Sun et al, 2019;, pruning (Michel et al, 2019;Chen et al, 2020;, and quantization (Zafrir et al, 2019;Bai et al, 2020;Shen et al, 2020). Among all these compression techniques, quantization is a popular solution as it still preserves the original network architecture.…”

“…Concretely, quantization aims to replace the 32-bit parameter with a low-bitwidth representation. In NLP, the recent Binary-BERT (Bai et al, 2020) pushes BERT quantization to the limit by weight binarization, and introduces quantization on activations to bring additional energy savings. However, compared with the one-hot tokens in language, the input image patches in vision contain richer information, and it is unclear whether the binarization is still effective in ViTs.…”