Speech to Text Adaptation: Towards an Efficient Cross-Modal Distillation

Abstract

Speech is one of the most effective means of communication and is full of information that helps the transmission of utterer's thoughts. However, mainly due to the cumbersome processing of acoustic features, phoneme or word posterior probability has frequently been discarded in understanding the natural language. Thus, some recent spoken language understanding (SLU) modules have utilized an end-to-end structure that preserves the uncertainty information. This further reduces the propagation of speech recognition error and guarantees computational efficiency. We claim that in this process, the speech comprehension can benefit from the inference of massive pretrained language models (LMs). We transfer the knowledge from a concrete Transformer-based text LM to an SLU module which can face a data shortage, based on recent cross-modal distillation methodologies. We demonstrate the validity of our proposal upon the performance on the Fluent Speech Command dataset. Thereby, we experimentally verify our hypothesis that the knowledge could be shared from the top layer of the LM to a fully speech-based module, in which the abstracted speech is expected to meet the semantic representation.

St-Bert: Cross-Modal Language Model Pre-Training for End-to-End Spoken Language Understanding

“…Data Shortage Scenario. To examine the robustness of model performance to varying training data size, we test our model with a small amount of data as presented in [2,5]. In Table 1, we observe a comparatively marginal performance degradation in ST-BERT for both of data shortage scenarios.…”

“…Chung et al [17] learn audio segment representations and word representations individually and aligns their spaces via adversarial training. Other works [5,6] match sequence-level representations of the two modalities using knowledge distillation [18] from a text encoder to a speech encoder. Speech-BERT [19] jointly trains multi-modal representations.…”

“…This allows the models to fully exploit additional information such as emotion and nuance characterized with acoustic signals. Recently, leveraging large-scale pre-trained language models (PLMs) such as BERT [4] has enhanced SLU performances [5,6] by benefiting from richly learned textual representation. However, these methods exploit only limited textual information by explicitly aligning the spoken utterance and its transcript representations.…”

Two-Stage Textual Knowledge Distillation for End-to-End Spoken Language Understanding

“…Several works use cross-modal distillation approach on SLU [13,14] to exploit textual knowledge. Cho et al [13] use knowledge distillation from a fine-tuned text BERT to an SLU model by making predicted logits for intent classification close to each other in fine-tuning. Denisov and Vu [14] match an utterance embedding and a sentence embeddings of ASR pairs using knowledge distillation as a pre-training.…”

“…Test accuracy on FSC almost reaches 100%, implying that there is little room to improve and evaluate the newly proposed method's effectiveness. Following [6,13], we simulate a data shortage scenario using only 10% of the speech-text pairs in training. We randomly divide FSC dataset into ten parts and report the average accuracy on them.…”