Efficient Intent Detection with Dual Sentence Encoders

Abstract

Building conversational systems in new domains and with added functionality requires resource-efficient models that work under lowdata regimes (i.e., in few-shot setups). Motivated by these requirements, we introduce intent detection methods backed by pretrained dual sentence encoders such as USE and Con-veRT. We demonstrate the usefulness and wide applicability of the proposed intent detectors, showing that: 1) they outperform intent detectors based on fine-tuning the full BERT-Large model or using BERT as a fixed black-box encoder on three diverse intent detection data sets; 2) the gains are especially pronounced in few-shot setups (i.e., with only 10 or 30 annotated examples per intent); 3) our intent detectors can be trained in a matter of minutes on a single CPU; and 4) they are stable across different hyperparameter settings. In hope of facilitating and democratizing research focused on intention detection, we release our code, as well as a new challenging single-domain intent detection dataset comprising 13,083 annotated examples over 77 intents.

ConveRT: Efficient and Accurate Conversational Representations from Transformers

“…Note that, besides quicker pretraining, intent classifiers based on ConveRT encodings train 40 times faster than BERT-LARGE-based ones, as only the classification layers are trained for ConveRT. Additional experiments related to efficiency of intent classification have been conducted by Casanueva et al (2020). In sum, these preliminary results suggest that ConveRT as a sentence encoder can be useful beyond the core response selection task.…”

“…In sum, these preliminary results suggest that ConveRT as a sentence encoder can be useful beyond the core response selection task. The usefulness of ConveRT-based sentence representations have been recently confirmed on other intent classification datasets (Casanueva et al, 2020), with different intent classifiers (Bunk et al, 2020), and in another dialog task: turn-based value extraction (Coope et al, 2020;Bunk et al, 2020;Mehri et al, 2020). In future work, we plan to investigate other possible applications of transfer, especially for the challenging low-data setups.…”

“…This notable reduction in size and training acceleration are achieved through combining 8-bit embedding quantization and quantization-aware training, subword-level parameterization, and pruned self-attention. Furthermore, the lightweight design allows us to reserve additional parameters to improve the expressiveness of the dual-encoder architecture; this leads to improved learning of conversational representations that can be transferred to other dialog tasks such as intent detection and slot filling, as already demonstrated by recent work (Casanueva et al, 2020;Bunk et al, 2020;Coope et al, 2020).…”

“…Therefore, we also probe the usefulness of ConveRT encodings for transfer learning in the intent classification task: the model must classify the user's utterance into one of several predefined classes, that is, intents (e.g., within e-banking intents can be card lost or replace card). We use BANKING77 (Casanueva et al, 2020) diverse domains, see Table 1, divided into train, dev and test sets using a 80/10/10 split. We use the pretrained ConveRT encodings r x on the input side (see Figure 1) as input to an intent classification model.…”

Improving Out-of-Scope Detection in Intent Classification by Using Embeddings of the Word Graph Space of the Classes

“…Notice that the intent classifier is typically implemented using standard text classification algorithms (Weiss et al, 2012;Larson et al, 2019;Casanueva et al, 2020). Consequently, to perform OOS sample detection, methods often rely on one-class classification or threshold rejectionbased techniques using the probability outputs for each class (Larson et al, 2019) or reconstruction errors (Ryu et al, 2017(Ryu et al, , 2018.…”