Parallel Intent and Slot Prediction using MLB Fusion

Abstract

Intent and Slot Identification are two important tasks in Spoken Language Understanding (SLU). For a natural language utterance, there is a high correlation between these two tasks. A lot of work has been done on each of these using Recurrent-Neural-Networks (RNN), Convolution Neural Networks (CNN) and Attention based models. Most of the past work used two separate models for intent and slot prediction. Some of them also used sequence-to-sequence type models where slots are predicted after evaluating the utterance-level intent. In this work, we propose a parallel Intent and Slot Prediction technique where separate Bidirectional Gated Recurrent Units (GRU) are used for each task. We posit the usage of MLB (Multimodal Low-rank Bilinear Attention Network) fusion for improvement in performance of intent and slot learning. To the best of our knowledge, this is the first attempt of using such a technique on text based problems. Also, our proposed methods outperform the existing state-of-the-art results for both intent and slot prediction on two benchmark datasets

A survey of joint intent detection and slot-filling models in natural language understanding

“…Vector representation of frame [Ray et al 2018] Rare, OOV words Paraphrasing input utterances [Liu et al 2019b] Unidirectional information flow Memory network [Shen et al 2019a] Poor generalisation in deployment Sparse word embedding (prune useless words) [Ray et al 2019] Slots which take many values perform poorly Delexicalisation Language knowledge base, history context Attention over external knowledge base, multiturn history Implicit knowledge sharing between tasks BiLSTM, multi-task (DA) [Gupta et al 2019a] Speed Non-recurrent and label recurrent networks [Gupta et al 2019b] Multi-turn dialogue, using context Token attention, previous history Capturing intent-slot correlation Multi-head self attention, masked intent Poor generalisation BERT [Bhasin et al 2019] Learning joint distribution CNN, BiLSTM, cross-fusion, masking [Thi Do and Gaspers 2019] Lack of annotated data, flexibility Language transfer, multitasking, modularisation ] Key verb-slot correlation Key verb in features, BiLSTM, attention [Zhang and Wang 2019] Learning joint distribution Transformer architecture [Daha and Hewavitharana 2019] Efficient modelling of temporal dependency Character embedding and RNN [Dadas et al 2019] Lack of annotated data, small data sets Augmented data set Learning joint distribution Word embedding attention [E et al 2019] Learning joint distribution Bidirectional architecture, feedback Poor generalisation BERT encoding, multi-head self attention [Qin et al 2019] Weak influence of intent on slot Use intent prediction instead of summarised intent info in slot tagging [Gangadharaiah and Narayanaswamy 2019] Multi-intent samples Multi-label classification methods [Firdaus et al 2019] Multi-turn dialogue history, learning joint distribution RNN, CRF [Pentyala et al 2019] Optimal architecture BiLSTM, different architectures Non-recurrent model, transfer learning BERT, language transfer [Schuster et al 2019] Low resource languages Transfer methods with SLU test case [Okur et al 2019] Natural language Locate intent keywords, non-other slots [Xu et al 2020] Only good performance in one sub-task Joint intent/slot tagging, length variable attention [Bhasin et al 2020] Learning joint distribution Multimodal Low-rank Bilinear Attention Network [Firdaus et al 2020] Learning joint distribution Stacked BiLSTM [Zhang et al 2020b] Limitations of sequential analysis Graph representation of text Non-convex optimisation Convex combination of ensemble of models BERT issues with logical d...…”

“…Importing methods from analogous fields. [Bhasin et al 2020] propose an interesting analogy; that the relationship between intent and slots is similar to that between the query and image in visual question answering. Thus they borrow an idea from the latter field -Multimodal Low Rank Bilinear (MLB) fusion, between the features of each part.…”

“…The gamut of word embedding methods have been used including word2vec ( [Pan et al 2018;Wang et al 2018c]), fastText ([Firdaus et al 2020]), GloVe ([Bhasin et al 2019;Bhasin et al 2020;Dadas et al 2019;Liu et al 2019b;Okur et al 2019;Pentyala et al 2019;Thi Do and Gaspers 2019;Zhang and Wang 2016]), ELMo [Zhang et al 2020b] and [Krone et al 2020] (pre-print only), BERT ([Ni et al 2020;Qin et al 2019;] and Krone et al 2020] (pre-print only) and [Han et al 2020] (submitted for publication). [Firdaus et al 2018a] and [Firdaus et al 2019] used concatenated GloVe and word2vec embeddings to capture more word information.…”