Subword Semantic Hashing for Intent Classification on Small Datasets

Abstract

In this paper, we introduce the use of Semantic Hashing as embedding for the task of Intent Classification and achieve state-of-the-art performance on three frequently used benchmarks. Intent Classification on a small dataset is a challenging task for data-hungry state-of-the-art Deep Learning based systems. Semantic Hashing is an attempt to overcome such a challenge and learn robust text classification. Current word embedding based methods [11], [13], [14] are dependent on vocabularies. One of the major drawbacks of such methods is out-of-vocabulary terms, especially when having small training datasets and using a wider vocabulary. This is the case in Intent Classification for chatbots, where typically small datasets are extracted from internet communication. Two problems arise with the use of internet communication. First, such datasets miss a lot of terms in the vocabulary to use word embeddings efficiently. Second, users frequently make spelling errors. Typically, the models for intent classification are not trained with spelling errors and it is difficult to think about ways in which users will make mistakes. Models depending on a word vocabulary will always face such issues. An ideal classifier should handle spelling errors inherently. With Semantic Hashing, we overcome these challenges and achieve state-of-the-art results on three datasets: Chatbot, Ask Ubuntu, and Web Applications [3]. Our benchmarks are available online. 1

Transformer-Capsule Model for Intent Detection (Student Abstract)

“…On AskUbuntu and WebApp we compared our solution with popular architectures 1 as well as current state-of-theart solution on those datasets -Subword Semantic Hashing (SSH) (Shridhar et al 2018 We also compared our solution with the original capsule implementation (Zhang et al 2018) that was tested on the ATIS dataset as well as the current SOTA on this dataset (Chen and Yu 2019 The results show that our solution is able to achieve stateof-the-art results on datasets with small number of examples, such as AskUbuntu and WebApp, as well as larger datasets like ATIS.…”

“…We have tested our solution on AskUbuntu and WebApp where we achieved results of 89% and 92%, outperforming current state-of-the-art solution (Shridhar et al 2018), as well as on ATIS dataset where we achieved 98.89% with current state-of-the-art (Chen and Yu 2019) at 98.61%.…”

A survey of joint intent detection and slot-filling models in natural language understanding

“…Adversarial training method for the multi-task and multi-lingual joint modelling [Mohasseb et al 2018] Grammar feature exploration Grammar-based framework with 3 main features [Xie et al 2018] Short text; Semantic feature expansion Semantic Tag-empowered combined features [Qiu et al 2018] Potential consciousness information mining A similarity calculation method based on LSTM and a traditional machine learning method based on multi-feature extraction OOD utterances Multi-task learning [Cohan et al 2019] Utilisation of naturally labelled data Multitask learning based on joint loss [Shridhar et al 2019] OOV issue; Small/lack of labelled training data Subword semantic hashing ] Learning of deep semantic information Hybrid CNN and bidirectional GRU neural network with pretrained embeddings (Char-CNN-BGRU) [Lin and Xu 2019] Emerging intents detection Maximise inter-class variance and minimise intra-class variance to get the discriminative feature [Ren and Xue 2020] Similar utterance with different intent Triples of samples used for training [Yilmaz and Toraman 2020] OOD utterances KL divergence vector for classification [Costello et al 2018] developed a novel multi-layer ensembling approach that ensembles both different model initialisation and different model architectures to determine how multi-layer ensembling improves performance on multilingual intent classification. They constructed a CNN with character-level embedding and a bidirectional CNN with attention mechanism.…”

Intent Detection-Based Lithuanian Chatbot Created via Automatic DNN Hyper-Parameter Optimization

“…The investigation reveals that IBM Watson significantly outperforms other platforms as Dialogflow, MS LUIS and Rasa that also demonstrate very good results. Three English benchmark datasets, i.e., askUbuntu, chatbot and webApps [8] were used in the experiments [9]. Authors introduce a sub-word semantic hashing technique to process input texts before classification.…”