Spoken Language Understanding on the Edge

Saade, Alaa; Dureau, Joseph; Luque, David; Caltagirone, Francesco; Coucke, Alice; Ball, Adrien; Doumouro, Clément; Lavril, Thibaut; Caulier, Alexandre; Bluche, Théodore; Gisselbrecht, Thibault; Primet, Maël

doi:10.1109/emc2-nips53020.2019.00021

Cited by 25 publications

(10 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The free audio-based Snips SLU Dataset [30] is quite close to what we want. This dataset has a great variety of phrases, but less audio-2,946 English utterances and 1,138 French utterances, compared to 30,043 in Fluent Speech Commands.…”

Section: Related Datasetssupporting

confidence: 57%

Speech Model Pre-Training for End-to-End Spoken Language Understanding

Lugosch

Ravanelli

Ignoto³

et al. 2019

Interspeech 2019

191

268

View full text Add to dashboard Cite

Whereas conventional spoken language understanding (SLU) systems map speech to text, and then text to intent, end-toend SLU systems map speech directly to intent through a single trainable model. Achieving high accuracy with these end-toend models without a large amount of training data is difficult. We propose a method to reduce the data requirements of endto-end SLU in which the model is first pre-trained to predict words and phonemes, thus learning good features for SLU. We introduce a new SLU dataset, Fluent Speech Commands, and show that our method improves performance both when the full dataset is used for training and when only a small subset is used. We also describe preliminary experiments to gauge the model's ability to generalize to new phrases not heard during training.

show abstract

Section: Related Datasetssupporting

confidence: 57%

Speech Model Pre-Training for End-to-End Spoken Language Understanding

Lugosch

Ravanelli

Ignoto³

et al. 2019

Interspeech 2019

191

268

View full text Add to dashboard Cite

show abstract

“…An example would be, projecting the acoustic and text-only data into the same embedding space, and learning the parameters of this joint space to optimize the SLU task-loss. This enables us to train an SLU model in an 1 For the dataset in [5] and [20] in order. end-to-end fashion on the smaller E2E dataset, but also leverage the large amount of ASR-or text-only datasets to learn a robust latent space for the final task.…”

Section: Cross-modal Latent Spacesmentioning

confidence: 99%

“…We use two publicly available SLU datasets to train and evaluate our model -Fluent Speech Commands (FSC) [5] and Snips SLU SmartLights [20]. Both these datasets contain utterance text, corresponding audio and a semantic class label.…”

Section: Datasetsmentioning

confidence: 99%

“…We train the so-called cross-modal latent space (CMLS) E2E SLU model architecture defined in [13] on two publicly available SLU datasets [5,20] and study the effect of these cross-modal embedding losses and of the adversarial training on the intent accuracy of the model. We show that the triplet loss has the best performance across both these datasets.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Tie Your Embeddings Down: Cross-Modal Latent Spaces for End-to-end Spoken Language Understanding

Agrawal¹,

Müller²,

Radfar³

et al. 2020

Preprint

View full text Add to dashboard Cite

End-to-end (E2E) spoken language understanding (SLU) systems can infer the semantics of a spoken utterance directly from an audio signal. However, training an E2E system remains a challenge, largely due to the scarcity of paired audio-semantics data. In this paper, we treat an E2E system as a multi-modal model, with audio and text functioning as its two modalities, and use a cross-modal latent space (CMLS) architecture, where a shared latent space is learned between the 'acoustic' and 'text' embeddings. We propose using different multi-modal losses to explicitly guide the acoustic embeddings to be closer to the text embeddings, obtained from a semantically powerful pre-trained BERT model. We train the CMLS model on two publicly available E2E datasets, across different cross-modal losses and show that our proposed triplet loss function achieves the best performance. It achieves a relative improvement of 1.4% and 4% respectively over an E2E model without a cross-modal space and a relative improvement of 0.7% and 1% over a previously published CMLS model using L2 loss. The gains are higher for a smaller, more complicated E2E dataset, demonstrating the efficacy of using an efficient cross-modal loss function, especially when there is limited E2E training data available.

show abstract

“…To address this, we present a dataset-agnostic framework for evaluating end-to-end model on decomposable tasks. Using spoken intent prediction as a case study, we focus on two popular benchmarks, FSC [4] and Snips SmartLights (Snips) [20,21] datasets. We provide evidence that the original test splits do not fairly evaluate the ASR and NLU subtasks of spoken intent prediction.…”

Section: Introductionmentioning

confidence: 99%

Rethinking End-to-End Evaluation of Decomposable Tasks: A Case Study on Spoken Language Understanding

Arora¹,

Ostapenko²,

Viswanathan³

et al. 2021

Preprint

View full text Add to dashboard Cite

Decomposable tasks are complex and comprise of a hierarchy of sub-tasks. Spoken intent prediction, for example, combines automatic speech recognition and natural language understanding. Existing benchmarks, however, typically hold out examples for only the surface-level sub-task. As a result, models with similar performance on these benchmarks may have unobserved performance differences on the other sub-tasks. To allow insightful comparisons between competitive end-to-end architectures, we propose a framework to construct robust test sets using coordinate ascent over sub-task specific utility functions. Given a dataset for a decomposable task, our method optimally creates a test set for each sub-task to individually assess sub-components of the end-to-end model. Using spoken language understanding as a case study, we generate new splits for the Fluent Speech Commands and Snips SmartLights datasets. Each split has two test sets: one with held-out utterances assessing natural language understanding abilities, and one with heldout speakers to test speech processing skills. Our splits identify performance gaps up to 10% between end-to-end systems that were within 1% of each other on the original test sets. These performance gaps allow more realistic and actionable comparisons between different architectures, driving future model development. We release our splits and tools for the community. 1

show abstract

Spoken Language Understanding on the Edge

Cited by 25 publications

References 10 publications

Speech Model Pre-Training for End-to-End Spoken Language Understanding

Speech Model Pre-Training for End-to-End Spoken Language Understanding

Tie Your Embeddings Down: Cross-Modal Latent Spaces for End-to-end Spoken Language Understanding

Rethinking End-to-End Evaluation of Decomposable Tasks: A Case Study on Spoken Language Understanding

Contact Info

Product

Resources

About