schuBERT: Optimizing Elements of BERT

Abstract

Transformers (Vaswani et al., 2017) have gradually become a key component for many state-of-the-art natural language representation models. A recent Transformer based model- BERT (Devlin et al., 2018) achieved state-of-the-art results on various natural language processing tasks, including GLUE, SQuAD v1.1, and SQuAD v2.0. This model however is computationally prohibitive and has a huge number of parameters. In this work we revisit the architecture choices of BERT in efforts to obtain a lighter model. We focus on reducing the number of parameters yet our methods can be applied towards other objectives such FLOPs or latency. We show that much efficient light BERT models can be obtained by reducing algorithmically chosen correct architecture design dimensions rather than reducing the number of Transformer encoder layers. In particular, our schuBERT gives 6.6% higher average accuracy on GLUE and SQuAD datasets as compared to BERT with three encoder layers while having the same number of parameters.

Compressing Large-Scale Transformer-Based Models: A Case Study on BERT

“…Mao et al (2020) combined distillation with unstructured pruning, while Hou et al (2020) combined distillation with structured pruning. When compared with only structured pruning (Khetan and Karnin, 2020), we see that Hou et al (2020) achieved both a smaller model size (12.4%) and also a smaller drop in accuracy (0.96%).…”

“…However, this is not a completely fair comparison, as Sanh et al (2019) did not use attention as a distillation target. When we compare other methods, we find that Jiao et al (2020) was able to beat Khetan and Karnin (2020) in terms of both model size and accuracy. This shows that structured pruning outperforms student models trained using distillation only on encoder outputs and output logits, but fails against distillation on attention maps.…”

“…Similarly to encoder unit pruning, we can reduce the size of the embedding vector (H) by pruning along the width of the model. Such a model can be obtained by either training with adaptive width, so that the model is robust to such pruning during inference (Hou et al, 2020), or by removing the least important feature dimensions iteratively (Khetan and Karnin, 2020;Prasanna et al, 2020;Tsai et al, 2020;.…”

“…As discussed in Section 3, structured pruning removes architectural components from BERT, which can also be seen as reducing the number of hyper-parameters that govern the BERT architecture. While pruned the encoder units (L) and reduced the model depth by half with an average accuracy drop of 1.0%, Khetan and Karnin (2020) took it a step further and systematically reduced both the depth (L) as well as the width (H, A) of the model, compressing to 39.1% of the original size with an average accuracy drop of only 1.86%. Detailed experiments by Khetan and Karnin (2020) also show that reducing all hyper-parameters in harmony, instead of focusing on just one, yields better performance.…”

“…However, some layers might be able to handle more compression. Methods compressing each layer independently (Khetan and Karnin, 2020;Tsai et al, 2020) have shown promising results, but remain under-explored.…”