Tie Your Embeddings Down: Cross-Modal Latent Spaces for End-to-end Spoken Language Understanding

Abstract

End-to-end (E2E) spoken language understanding (SLU) systems can infer the semantics of a spoken utterance directly from an audio signal. However, training an E2E system remains a challenge, largely due to the scarcity of paired audio-semantics data. In this paper, we treat an E2E system as a multi-modal model, with audio and text functioning as its two modalities, and use a cross-modal latent space (CMLS) architecture, where a shared latent space is learned between the 'acoustic' and 'text' embeddings. We propose using different multi-modal losses to explicitly guide the acoustic embeddings to be closer to the text embeddings, obtained from a semantically powerful pre-trained BERT model. We train the CMLS model on two publicly available E2E datasets, across different cross-modal losses and show that our proposed triplet loss function achieves the best performance. It achieves a relative improvement of 1.4% and 4% respectively over an E2E model without a cross-modal space and a relative improvement of 0.7% and 1% over a previously published CMLS model using L2 loss. The gains are higher for a smaller, more complicated E2E dataset, demonstrating the efficacy of using an efficient cross-modal loss function, especially when there is limited E2E training data available.

Speak or Chat with Me: End-to-End Spoken Language Understanding System with Flexible Inputs

“…We obtain significant improvements in performance over the baseline model on the Snips [18] and Fluent Speech Commands (FSC) datasets [10]. To welcome researchers to improve upon our work similar to [10] [14], we are releasing our codebase. 1…”

“…Alternatively, authors in [13] successfully co-train a text-to-intent (T2I) model and speech-to-intent (S2I) model to closely align the acoustic embeddings with BERT-based text embeddings. Further improving on this cross-modal approach, authors in [14] employ the triplet loss function to learn a robust cross-modal latent space. Training the acoustic and text embeddings in the shared latent space makes it easier to combine separate audio and text data.…”

“…This also allows us to produce an intent prediction for a chat transcript without the audio counterpart. We choose the cross-modal system in [14] as our baseline model and further investigate and improve upon their approach. Their model consists of two branches: an acoustic branch that takes audio input, and a BERT text branch that takes the ground truth transcripts of the audio files.…”

“…Their model consists of two branches: an acoustic branch that takes audio input, and a BERT text branch that takes the ground truth transcripts of the audio files. Triplet loss ties the acoustics embeddings and text embeddings together, thus forcing samples in the same class to be closer in the feature space and pushing samples from different classes to be further apart [14].…”

Improving End-to-End Models for Set Prediction in Spoken Language Understanding

“…SLU systems have traditionally been a cascade of an automatic speech recognition (ASR) system converting speech into text followed by a natural language understanding (NLU) system that interprets the meaning of the text [1][2][3][4]. In contrast, an end-to-end (E2E) SLU system [5][6][7][8][9][10][11][12][13][14] processes speech input directly into meaning without going through an intermediate text transcript.…”