Two-Stage Textual Knowledge Distillation for End-to-End Spoken Language Understanding

Abstract

End-to-end approaches open a new way for more accurate and efficient spoken language understanding (SLU) systems by alleviating the drawbacks of traditional pipeline systems. Previous works exploit textual information for an SLU model via pre-training with automatic speech recognition or finetuning with knowledge distillation. To utilize textual information more effectively, this work proposes a two-stage textual knowledge distillation method that matches utterancelevel representations and predicted logits of two modalities during pre-training and fine-tuning, sequentially. We use vq-wav2vec BERT as a speech encoder because it captures general and rich features. Furthermore, we improve the performance, especially in a low-resource scenario, with data augmentation methods by randomly masking spans of discrete audio tokens and contextualized hidden representations. Consequently, we push the state-of-the-art on the Fluent Speech Commands, achieving 99.7% test accuracy in the full dataset setting and 99.5% in the 10% subset setting. Throughout the ablation studies, we empirically verify that all used methods are crucial to the final performance, providing the best practice for spoken language understanding. Code is available at https://github.com/clovaai/textual-kd-slu.

A Comprehensive Review of Knowledge Distillation- Methods, Applications, and Future Directions

“…By simultaneously optimizing text and image generation models, they improved the quality and consistency of image generation. Kim et al introduced a knowledge distillation method from speech to text, named Speech2Text Distillation [35] , leveraging pretrained speech recognition models to enhance text generation models. They significantly improved the performance of speech-to-text tasks through cross-modal distillation.…”