Stop: A Dataset for Spoken Task Oriented Semantic Parsing

Abstract

End-to-end spoken language understanding (SLU) predicts intent directly from audio using a single model. It promises to improve the performance of assistant systems by leveraging acoustic information lost in the intermediate textual representation and preventing cascading errors from Automatic Speech Recognition (ASR). Further, having one unified model has efficiency advantages when deploying assistant systems on-device. However, the limited number of public audio datasets with semantic parse labels hinders the research progress in this area. In this paper, we release the Spoken Task-Oriented semantic Parsing (STOP) dataset 1 , the largest and most complex SLU dataset publicly available. Additionally, we define low-resource splits to establish a benchmark for improving SLU when limited labeled data is available. Furthermore, in addition to the human-recorded audio, we are releasing a TTS-generated versions to benchmark the performance for low-resource and domain adaptation of end-to-end SLU systems.

Modality Confidence Aware Training for Robust End-to-End Spoken Language Understanding

“…We present two novel techniques to improve E2E SLU models: 1) a method to encode ASR hypothesis quality and 2) an effective method to integrate these quality information into E2E SLU models. We show accuracy improvements on STOP dataset [16] in the on-device streaming scenario and share the analysis to demonstrate the effectiveness of our approach.…”

“…We used the largest public SLU dataset, STOP (Spoken Task Oriented Semantic Parsing) [16] to evaluate our proposed approach. The STOP dataset is based on Task-Oriented Semantic Parsing (TOPv2) [25], a well-known NLU benchmark, that covers 8 different domains including alarm, messaging, music, navigation, timer, weather, reminder, and event.…”

HornBase: An audio dataset of car horns in different scenarios and positions

“…Additionally, audio datasets collected by the authors themselves in real environments or situations were observed. Examples include the Chime-Home [ 7 ], a dataset of gunshot audio [ 8 ], one focused on motor sounds [ 9 ], and some specific resources for spoken tasks, such as AudioMNIST [ 10 ] and STOP [ 11 ]. Also, there are audio datasets created through cutting, modifications, and transformations applied to existing datasets, such as SARdB [ 12 ] for audio scenes and Shrutilipi [ 13 ] for automatic speech recognition.…”