Knowledge Distillation from BERT Transformer to Speech Transformer for Intent Classification

Abstract

End-to-end intent classification using speech has numerous advantages compared to the conventional pipeline approach using automatic speech recognition (ASR), followed by natural language processing modules. It attempts to predict intent from speech without using an intermediate ASR module. However, such end-to-end framework suffers from the unavailability of large speech resources with higher acoustic variation in spoken language understanding. In this work, we exploit the scope of the transformer distillation method that is specifically designed for knowledge distillation from a transformer based language model to a transformer based speech model. In this regard, we leverage the reliable and widely used bidirectional encoder representations from transformers (BERT) model as a language model and transfer the knowledge to build an acoustic model for intent classification using the speech. In particular, a multilevel transformer based teacher-student model is designed, and knowledge distillation is performed across attention and hidden sub-layers of different transformer layers of the student and teacher models. We achieve an intent classification accuracy of 99.10% and 88.79% for Fluent speech corpus and ATIS database, respectively. Further, the proposed method demonstrates better performance and robustness in acoustically degraded condition compared to the baseline method.

New Avenues for Automated Railway Safety Information Processing in Enterprise Architecture: An NLP Approach

“…Its architecture is similar to BERT, but token embedding and pools are removed with several layers less by a factor of two. All other vital operations such as linear layer and layer normalisation are improved using model linear algebra frameworks [55]. Electra Small is another pre-trained transformer model introduced by Google.…”

“…For the remaining training dataset, sentence A is paired with random sentence B with the 'NotNext' label. This pre-training is very beneficial in Question Answering and Natural language Interface [55].…”

Mokey

“…Recent model compression works fall under three general classes: Pruning which forces some weights or activations to zero [20-22, 26, 32, 34, 40, 47] combined with "zero-aware" memory encoding. Knowledge distillation distills a larger, "teacher" model into a smaller "student" model [1,17,24,27,29,37,43]. Lastly, quantization where the parameters and/or activations are quantized to shorter bit-widths [6,19,39,50,51,54].…”