Hierarchical Multi-Task Natural Language Understanding for Cross-domain Conversational AI: HERMIT NLU

Vanzo, Andrea; Bastianelli, Emanuele; Lemon, Oliver

doi:10.18653/v1/w19-5931

“…System Descriptions: We evaluate SLURP against two state-of-the-art NLU models: HerMiT (Vanzo et al, 2019) and SF-ID (E et al, 2019). Both systems achieved state-of-the-art results on the NLU Benchmark (Liu et al, 2019) and on ATIS/Snips respectively.…”

Section: Semantic Evaluationmentioning

confidence: 99%

SLURP: A Spoken Language Understanding Resource Package

Bastianelli¹,

Vanzo²,

Swietojanski³

et al. 2020

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)

Self Cite

View full text Add to dashboard Cite

Spoken Language Understanding infers semantic meaning directly from audio data, and thus promises to reduce error propagation and misunderstandings in end-user applications. However, publicly available SLU resources are limited. In this paper, we release SLURP, a new SLU package containing the following: (1) A new challenging dataset in English spanning 18 domains, which is substantially bigger and linguistically more diverse than existing datasets; (2) Competitive baselines based on state-of-the-art NLU and ASR systems; (3) A new transparent metric for entity labelling which enables a detailed error analysis for identifying potential areas of improvement. SLURP is available at https:

show abstract

“…Joint training of Intent Recognition and Entity Extraction models have been explored recently (Zhang and Wang, 2016;Liu and Lane, 2016;Goo et al, 2018;Varghese et al, 2020). Several hierarchical multi-task architectures are proposed for these joint NLU approaches (Zhou et al, 2016;Wen et al, 2018;Okur et al, 2019;Vanzo et al, 2019), few of them in multimodal context (Gu et al, 2017;Okur et al, 2020). Vaswani et al (2017) proposed the Transformer as a novel neural network architecture based entirely on attention mechanisms (Bahdanau et al, 2015).…”

Section: Related Workmentioning

confidence: 99%

NLU for Game-based Learning in Real: Initial Evaluations

Okur¹,

Sahay²,

Nachman³

2022

Preprint

0

View full text Add to dashboard Cite

Intelligent systems designed for play-based interactions should be contextually aware of the users and their surroundings. Spoken Dialogue Systems (SDS) are critical for these interactive agents to carry out effective goal-oriented communication with users in real-time. For the real-world (i.e., in-the-wild) deployment of such conversational agents, improving the Natural Language Understanding (NLU) module of the goal-oriented SDS pipeline is crucial, especially with limited task-specific datasets. This study explores the potential benefits of a recently proposed transformer-based multi-task NLU architecture, mainly to perform Intent Recognition on small-size domain-specific educational game datasets. The evaluation datasets were collected from children practicing basic math concepts via play-based interactions in game-based learning settings. We investigate the NLU performances on the initial proof-of-concept game datasets versus the real-world deployment datasets and observe anticipated performance drops in-the-wild. We have shown that compared to the more straightforward baseline approaches, Dual Intent and Entity Transformer (DIET) architecture (Bunk et al., 2020) is robust enough to handle real-world data to a large extent for the Intent Recognition task on these domain-specific in-the-wild game datasets.

show abstract

“…System Descriptions: We evaluate SLURP against two state-of-the-art NLU models: HerMiT (Vanzo et al, 2019) and SF-ID (E et al, 2019). Both systems achieved state-of-the-art results on the NLU Benchmark (Liu et al, 2019) and on ATIS/Snips respectively.…”

Section: Semantic Evaluationmentioning

confidence: 99%

SLURP: A Spoken Language Understanding Resource Package

Bastianelli¹,

Vanzo²,

Swietojanski³

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

Spoken Language Understanding infers semantic meaning directly from audio data, and thus promises to reduce error propagation and misunderstandings in end-user applications. However, publicly available SLU resources are limited. In this paper, we release SLURP, a new SLU package containing the following: (1) A new challenging dataset in English spanning 18 domains, which is substantially bigger and linguistically more diverse than existing datasets; (2) Competitive baselines based on state-of-the-art NLU and ASR systems; (3) A new transparent metric for entity labelling which enables a detailed error analysis for identifying potential areas of improvement. SLURP is available at https: //github.com/pswietojanski/slurp

show abstract

Hierarchical Multi-Task Natural Language Understanding for Cross-domain Conversational AI: HERMIT NLU

Cited by 22 publications

References 28 publications

SLURP: A Spoken Language Understanding Resource Package

SLURP: A Spoken Language Understanding Resource Package

NLU for Game-based Learning in Real: Initial Evaluations

SLURP: A Spoken Language Understanding Resource Package

Contact Info

Product

Resources

About