GOBO: Quantizing Attention-Based NLP Models for Low Latency and Energy Efficient Inference

Zadeh, Ali Hadi; Edo, Isak; Awad, Omar Mohamed; Moshovos, Andreas

doi:10.1109/micro50266.2020.00071

Cited by 75 publications

(39 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In Figure 1 and Table 8, we compare the proposed TernaryBERT with (i) Other Quantization Methods: including mixed-precision Q-BERT (Shen et al, 2020), post-training quantization GOBO (Zadeh and Moshovos, 2020), as well as Quant-Noise which uses product quantization (Fan et al, 2020); and (ii) Other Compression Methods: including weight-sharing method ALBERT (Lan et al, 2019), pruning method LayerDrop (Fan et al, 2019), distillation methods DistilBERT and TinyBERT (Sanh et al, 2019;Jiao et al, 2019). The result of Distil-BERT is taken from (Jiao et al, 2019).…”

Section: Comparison With Other Methodsmentioning

confidence: 99%

“…On Transformer-based models, 8-bit fixedpoint quantization is successfully applied in fullyquantized Transformer (Prato et al, 2019) and Q8BERT (Zafrir et al, 2019). The use of lower bits is also investigated in (Shen et al, 2020;Fan et al, 2020;Zadeh and Moshovos, 2020). Specifically, In Q-BERT (Shen et al, 2020) and GOBO (Zadeh and Moshovos, 2020), mixed-precision with 3 or more bits are used to avoid severe accuracy drop.…”

Section: Quantizationmentioning

confidence: 99%

“…The use of lower bits is also investigated in (Shen et al, 2020;Fan et al, 2020;Zadeh and Moshovos, 2020). Specifically, In Q-BERT (Shen et al, 2020) and GOBO (Zadeh and Moshovos, 2020), mixed-precision with 3 or more bits are used to avoid severe accuracy drop. However, mixed-precision quantization can be unfriendly to some hardwares.…”

Section: Quantizationmentioning

confidence: 99%

“…However, quantizing these models to ultra low bits (e.g., 1 or 2 bits) can be much more challenging due to significant reduction in model capacity. To avoid severe accuracy drop, more complex quantization methods, like mixed-precision quantization (Shen et al, 2020;Zadeh and Moshovos, 2020) and product quantization (PQ) (Fan et al, 2020), are used. However, mixed-precision quantization is unfriendly to some hardwares, and PQ requires extra clustering operations.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

TernaryBERT: Distillation-aware Ultra-low Bit BERT

Zhang¹,

Hou²,

Yin³

et al. 2020

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)

View full text Add to dashboard Cite

Transformer-based pre-training models like BERT have achieved remarkable performance in many natural language processing tasks. However, these models are both computation and memory expensive, hindering their deployment to resource-constrained devices. In this work, we propose TernaryBERT, which ternarizes the weights in a fine-tuned BERT model. Specifically, we use both approximation-based and loss-aware ternarization methods and empirically investigate the ternarization granularity of different parts of BERT. Moreover, to reduce the accuracy degradation caused by the lower capacity of low bits, we leverage the knowledge distillation technique (Jiao et al., 2019) in the training process. Experiments on the GLUE benchmark and SQuAD show that our proposed TernaryBERT outperforms the other BERT quantization methods, and even achieves comparable performance as the fullprecision model while being 14.9x smaller.

show abstract

Section: Comparison With Other Methodsmentioning

confidence: 99%

Section: Quantizationmentioning

confidence: 99%

Section: Quantizationmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

TernaryBERT: Distillation-aware Ultra-low Bit BERT

Zhang¹,

Hou²,

Yin³

et al. 2020

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)

View full text Add to dashboard Cite

show abstract

“…However, this method collects information from surrounding pixels and cannot generate dense contextual information. In recent years, attention modules have been successful in fields such as natural language processing [7][8][9][10][11], speech recognition [12,13], image inpainting [14][15][16] and image recognition [17][18][19]. The self-attention layer of work [8,11] This is an open access article under the terms of the Creative Commons Attribution License, which permits use, distribution and reproduction in any medium, provided the original work is properly cited.…”

Section: Introductionmentioning

confidence: 99%

Bilateral attention network for semantic segmentation

Wang

Zhou

et al. 2021

IET Image Processing

View full text Add to dashboard Cite

Enhancing network feature representation capabilities and reducing the loss of image details have become the focus of semantic segmentation task. This work proposes the bilateral attention network for semantic segmentation. The authors embed two attention modules in the encoder and decoder structures . Specifically, high-level features of the encoder structure integrate all channel maps through dense channel relationships learned by the channel correlation coefficient attention module. The positively correlated channels promote each other, and the negatively correlated channels suppress each other. In the decoder structure, low-level features selectively emphasize the edge detail information in the feature map through the position attention module. The feature expression of semantic segmentation is improved by feature fusion of the two attention modules to obtain more accurate segmentation results . Finally, to verify the effectiveness of the model, the authors conduct experiments on the PASCAL VOC 2012 and Cityscapes scene analysis benchmark data sets and achieve a mean intersection-over-union of 74.92% and 66.63%, respectively.

show abstract