Transfer Learning for Context-Aware Spoken Language Understanding

Abstract

Spoken language understanding (SLU) is a key component of task-oriented dialogue systems. SLU parses natural language user utterances into semantic frames. Previous work has shown that incorporating context information significantly improves SLU performance for multi-turn dialogues. However, collecting a large-scale human-labeled multi-turn dialogue corpus for the target domains is complex and costly. To reduce dependency on the collection and annotation effort, we propose a Context Encoding Language Transformer (CELT) model facilitating exploiting various context information for SLU. We explore different transfer learning approaches to reduce dependency on data collection and annotation. In addition to unsupervised pre-training using large-scale general purpose unlabeled corpora, such as Wikipedia, we explore unsupervised and supervised adaptive training approaches for transfer learning to benefit from other in-domain and out-ofdomain dialogue corpora. Experimental results demonstrate that the proposed model with the proposed transfer learning approaches achieves significant improvement on the SLU performance over state-of-the-art models on two large-scale single-turn dialogue benchmarks and one large-scale multiturn dialogue benchmark.

Exploring Bert-Based Encoders for Sequence Classification and Multi-Task Learning in Dialogue Acts and Joint Intent-Slot Filling

“…The BERT model [Devlin (2019)] and its derivatives have been fine-tuned for dialogue act classification, achieving state-of-the-art results by leveraging deep bidirectional representations. Furthermore, the JointBERT model [Chen (2019)] combines intent detection and slot filling with dialogue act classification, showcasing the power of multi-task learning.…”

Adaptive Global-Local Context Fusion for Multi-Turn Spoken Language Understanding

“…(2013) found that using previous utterances as contexts in an SVM-HMM SLU system could help resolve ambiguities. Recently, deep learning approaches have become increasingly popular to incorporate contextual information (Qin et al 2021;Su, Yuan, and Chen 2019;Abro et al 2019;Chen et al 2019;Su, Yuan, and Chen 2018;Gupta, Rastogi, and Hakkani-Tur 2018;Chen et al 2016;Wei et al 2021). proposed to use end-to-end memory networks to model previous utterance transcripts in multi-turn dialogues.…”

Attentive Contextual Carryover for Multi-Turn End-to-End Spoken Language Understanding

“…Contexts have been shown to significantly improve performance separately for ASR [7][8][9][10][11][12][13][14][15] and NLU [11,[16][17][18][19][20][21]. -Dialogue Act: [(a=REQUEST, s=Item), a=REQUEST, s=Quantity)] Fig.…”