Investigation of recurrent-neural-network architectures and learning methods for spoken language understanding

Mesnil, Grégoire; He, Xiaodong; Deng, Li; Bengio, Yoshua

doi:10.21437/interspeech.2013-596

Cited by 310 publications

(156 citation statements)

References 16 publications

Supporting

Mentioning

154

Contrasting

Unclassified

Order By: Relevance

“…RNNs with different architectures have been explored in many studies, considering their promising performance in sequence modeling elsewhere. In 2013, [Mesnil et al 2013] compared recurrent neural networks, including Elman-type and Jordan-type networks and bi-directional Jordan-type RNNs. Two years later, [Mesnil et al 2015] implemented Elman-type and Jordan-type networks and also their variations.…”

Section: 21mentioning

confidence: 99%

“…Deep belief network (DBN) [Mesnil et al 2013] To explore RNN RNN [Yao et al 2013] To explore RNN-LM RNN-LM [Yao et al 2014b] Label dependencies, label bias problem RNN, CRF [Yao et al 2014a] Gradient diminishing and exploding problem, label dependencies, label bias problem LSTM, regression model, deep learning [Liu and Lane 2015] Label dependencies RNN, sampling approach [Mesnil et al 2015] To explore RNN RNN Vanishing and exploding gradient RNN, external memory [Kurata et al 2016] Label dependencies LSTM, encoder-labeler To explore past and future information Bi-directional RNN, ranking loss function [Vu 2016] To explore CNN CNN [Zhu and Yu 2017] To explore attention mechanism Bi-directional LSTM, LSTM, encoder-decoder, focus mechanism [Dai et al 2018] Unseen slots CRF [Gong et al 2019] To explore MTL MTL, segment tagging, NER [Louvan and Magnini 2018] To explore MTL MTL, NER, bi-LSTM, CRF [Shin et al 2018] To better labelling common words Encoder-decoder attention, delexicalised sentence generation [Wang et al 2018a] Imbalanced data DNN, reinforcement learning [Zhao and Feng 2018] OOV GRU, attention, pointer network [Gong et al 2019] To explore MTL MTL, segment tagging, NER To extend original SLU to H2H conversations Bi-LSTM, different knowledge sources [Shen et al 2019b…”

Section: Papermentioning

confidence: 99%

See 1 more Smart Citation

A survey of joint intent detection and slot-filling models in natural language understanding

Weld¹,

Huang²,

Long³

et al. 2021

Preprint

View full text Add to dashboard Cite

Intent classification and slot filling are two critical tasks for natural language understanding. Traditionally the two tasks have been deemed to proceed independently. However, more recently, joint models for intent classification and slot filling have achieved state-of-the-art performance, and have proved that there exists a strong relationship between the two tasks. This article is a compilation of past work in natural language understanding, especially joint intent classification and slot filling. We observe three milestones in this research so far: Intent detection to identify the speaker's intention, slot filling to label each word token in the speech/text, and finally, joint intent classification and slot filling tasks. In this article, we describe trends, approaches, issues, data sets, evaluation metrics in intent classification and slot filling. We also discuss representative performance values, describe shared tasks, and provide pointers to future work, as given in prior works. To interpret the state-of-the-art trends, we provide multiple tables that describe and summarise past research along different dimensions, including the types of features, base approaches, and dataset domain used.

show abstract

Section: 21mentioning

confidence: 99%

Section: Papermentioning

confidence: 99%

A survey of joint intent detection and slot-filling models in natural language understanding

Weld¹,

Huang²,

Long³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…The majority of works on task-oriented semantic parsing focused on non-compositional user queries (Mesnil et al, 2013;Liu and Lane, 2016;Goo et al, 2018;, which turns the parsing task into a combination of intent detection and slot filling. Recently, Gupta et al (2018) Figure 3: The architecture of X2Parser.…”

Section: Task-oriented Semantic Parsingmentioning

confidence: 99%

X2Parser: Cross-Lingual and Cross-Domain Framework for Task-Oriented Compositional Semantic Parsing

Liu¹,

Winata²,

Xu³

et al. 2021

Preprint

View full text Add to dashboard Cite

Task-oriented compositional semantic parsing (TCSP) handles complex nested user queries and serves as an essential component of virtual assistants. Current TCSP models rely on numerous training data to achieve decent performance but fail to generalize to low-resource target languages or domains. In this paper, we present X2Parser, a transferable Crosslingual and Cross-domain Parser for TCSP. Unlike previous models that learn to generate the hierarchical representations for nested intents and slots, we propose to predict flattened intents and slots representations separately and cast both prediction tasks into sequence labeling problems. After that, we further propose a fertility-based slot predictor that first learns to dynamically detect the number of labels for each token, and then predicts the slot types. Experimental results illustrate that our model can significantly outperform existing strong baselines in cross-lingual and cross-domain settings, and our model can also achieve a good generalization ability on target languages of target domains. Furthermore, our model tackles the problem in an efficient nonautoregressive way that reduces the latency by up to 66% compared to the generative model. 1

show abstract

“…Because slot filling requires token-level annotations of semantic frame, while these methods can only provide sentence-level labels. Spoken Language understanding, including slot filling and intent detection tasks, has drawn a lot of research attention recently (Yao et al 2013(Yao et al , 2014Mesnil et al 2013Mesnil et al , 2015Chen et al 2016a,b;Goo et al 2018;Haihong et al 2019;Liu et al 2019). In this paper, we only focus on the slot filling task.…”

Section: Related Workmentioning

confidence: 99%

C2C-GenDA: Cluster-to-Cluster Generation for Data Augmentation of Slot Filling

Hou¹,

Chen²,

Che³

et al. 2020

Preprint

View full text Add to dashboard Cite

Slot filling, a fundamental module of spoken language understanding, often suffers from insufficient quantity and diversity of training data. To remedy this, we propose a novel Cluster-to-Cluster generation framework for Data Augmentation (DA), named C2C-GenDA. It enlarges the training set by reconstructing existing utterances into alternative expressions while keeping semantic. Different from previous DA works that reconstruct utterances one by one independently, C2C-GenDA jointly encodes multiple existing utterances of the same semantics and simultaneously decodes multiple unseen expressions. Jointly generating multiple new utterances allows to consider the relations between generated instances and encourages diversity. Besides, encoding multiple existing utterances endows C2C with a wider view of existing expressions, helping to reduce generation that duplicates existing data. Experiments on ATIS and Snips datasets show that instances augmented by C2C-GenDA improve slot filling by 7.99 (11.9%↑) and 5.76 (13.6%↑) F-scores respectively, when there are only hundreds of training utterances. 1 * Equal contributions.

show abstract

Investigation of recurrent-neural-network architectures and learning methods for spoken language understanding

Cited by 310 publications

References 16 publications

A survey of joint intent detection and slot-filling models in natural language understanding

A survey of joint intent detection and slot-filling models in natural language understanding

X2Parser: Cross-Lingual and Cross-Domain Framework for Task-Oriented Compositional Semantic Parsing

C2C-GenDA: Cluster-to-Cluster Generation for Data Augmentation of Slot Filling

Contact Info

Product

Resources

About