Semantic Complexity in End-to-End Spoken Language Understanding

Abstract

End-to-end spoken language understanding (SLU) models are a class of model architectures that predict semantics directly from speech. Because of their input and output types, we refer to them as speech-to-interpretation (STI) models. Previous works have successfully applied STI models to targeted use cases, such as recognizing home automation commands, however no study has yet addressed how these models generalize to broader use cases. In this work, we analyze the relationship between the performance of STI models and the difficulty of the use case to which they are applied. We introduce empirical measures of dataset semantic complexity to quantify the difficulty of the SLU tasks. We show that near-perfect performance metrics for STI models reported in the literature were obtained with datasets that have low semantic complexity values. We perform experiments where we vary the semantic complexity of a large, proprietary dataset and show that STI model performance correlates with our semantic complexity measures, such that performance increases as complexity values decrease. Our results show that it is important to contextualize an STI model's performance with the complexity values of its training dataset to reveal the scope of its applicability.

End-to-End Spoken Language Understanding for Generalized Voice Assistants

“…We use two E2E SLU datasets for our experiments -(1) the publicly available Fluent Speech Commands (FSC) and ( 2) an internal SLU dataset. Additionally, we create a "hard test set" to assess model performance in the most demanding scenarios in generalized VA. We use the average n-gram entropy and Minimum Spanning Tree (MST) complexity score as described in [27] to quantify their levels of semantic complexity. Fluent Speech Commands -FSC [21] is an SLU dataset containing 30,043 utterances with a vocabulary of 124 words and 248 unique utterances over 31 intents in home appliance and smart speaker control.…”

“…The SLU task on this dataset is just the intent classification task. It has an average n-gram entropy of 6.9 bits and an average MST complexity score of 0.2 [27].…”

“…Over the last year the state-of-the-art on FSC has progressed to over 99% test set accuracy for several E2E approaches [14][15][16][17][18][19][20]. However, there remains a gap between the capabilities demonstrated thus far and the E2E SLU requirements for a generalized VA [27]. In particular, existing benchmarks focus on tasks with limited semantic complexity and output structural diversity.…”

“…This leads to tasks with a long tail of rare utterances containing unique n-grams and specific slot values unseen during training, that are more semantically complex than the tasks tackled in aforementioned benchmark SLU datasets. Differences in semantic complexity across datasets can be assessed using n-gram entropy and utterance embedding MST complexity measures [27]. Furthermore, in generalized VA tasks the output label space is countably infinite, as any arbitrary sequence of words could be a valid slot output.…”

Could a Conversational AI Identify Offensive Language?

“…A Conversational AI is composed by end-to-end spoken language understanding (SLU) models to predict semantics directly from speech [3]. The conventional approach to SLU uses two distinct components to sequentially process a spoken utterance: an automatic speech recognition (ASR) model that transcribes the speech to a text transcript, followed by a natural language understanding (NLU) model that predicts the domain, intent, and entities given the transcript.…”