User Utterance Acquisition for Training Task-Oriented Bots: A Review of Challenges, Techniques and Opportunities

Yaghoub-Zadeh-Fard, Mohammad-Ali; Benatallah, Boualem; Casati, Fabio; Barukh, Moshe Chai; Zamanirad, Shayan

doi:10.1109/mic.2020.2978157

Cited by 13 publications

(16 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To evaluate the model's OOD detection performance, we consider a standard OOD task using SVHN as the OOD dataset for a model trained on CIFAR-10/-100, and a difficult OOD task using CIFAR-100 as the OOD dataset for a model trained on CIFAR-10, and vice versa. Full experiment details and additional results are in Appendix C. Detecting Out-of-Scope Intent in Conversational Language Understanding To validate the method beyond image modalities, we also evaluate SNGP on a practical language understanding task where uncertainty quantification is of natural importance: dialog intent detection [43,76,79,81]. In a goal-oriented dialog system (e.g.…”

Section: Vision and Language Understandingmentioning

confidence: 99%

Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness

Liu¹,

Zi²,

Padhy³

et al. 2020

Preprint

View full text Add to dashboard Cite

Bayesian neural networks (BNN) and deep ensembles are principled approaches to estimate the predictive uncertainty of a deep learning model. However their practicality in real-time, industrial-scale applications are limited due to their heavy memory and inference cost. This motivates us to study principled approaches to high-quality uncertainty estimation that require only a single deep neural network (DNN). By formalizing the uncertainty quantification as a minimax learning problem, we first identify input distance awareness, i.e., the model's ability to quantify the distance of a testing example from the training data in the input space, as a necessary condition for a DNN to achieve high-quality (i.e., minimax optimal) uncertainty estimation. We then propose Spectral-normalized Neural Gaussian Process (SNGP), a simple method that improves the distance-awareness ability of modern DNNs, by adding a weight normalization step during training and replacing the output layer. On a suite of vision and language understanding tasks and on modern architectures (Wide-ResNet and BERT), SNGP is competitive with deep ensembles in prediction, calibration and out-of-domain detection, and outperforms the other single-model approaches.

show abstract

Section: Vision and Language Understandingmentioning

confidence: 99%

Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness

Liu¹,

Zi²,

Padhy³

et al. 2020

Preprint

View full text Add to dashboard Cite

show abstract

“…Recently natural language interfaces (also known as bots, chatbots, dialog systems, and virtual assistants) have attracted enormous attention because of their potentials in hands-free environments (e.g., self-driving cars) and accessibility products [ 1 ]. Building such interfaces often requires training utterances (e.g., book a flight from Sydney to Paris ), annotated with user intents (e.g., book-flight) and associated parameters 1 (e.g., from = “Sydney”, to = “Paris”) [ 1 , 2 ]. Training utterances are used to train supervised models for detecting the users' intent based on their utterances.…”

Section: Objectivementioning

confidence: 99%

API2CAN: a dataset & service for canonical utterance generation for REST APIs

Yaghoub-Zadeh-Fard

Benatallah

2021

BMC Res Notes

Self Cite

View full text Add to dashboard Cite

Objectives Recently natural language interfaces (e.g., chatbots) have gained enormous attention. Such interfaces execute underlying application programming interfaces (APIs) based on the user's utterances to perform tasks (e.g., reporting weather). Supervised approaches for building such interfaces rely upon a large set of user utterances paired with APIs. Collecting such pairs is typically starts with obtaining initial utterances for a given API method. Generating initial utterances can be considered as a machine translation task in which an API method is translated into an utterance. However, the key challenge is the lack of training samples for training domain-independent translation models. In this paper, we propose a dataset for training supervised models to generate initial utterances for APIs. Data description The dataset contains 14,370 pairs of API methods and utterances. It is built automatically by converting method descriptions of a large number of APIs to user utterances; and it is cleaned manually to ensure quality. The dataset is also accompanied with a set of microservices (e.g., translating API methods to utterances) which can facilitate the process of collecting training samples for building natural language interfaces.

show abstract

“…The pipeline supports the evaluation and experimentation of existing and novel pipelines by leveraging the configuration and extension features, as well as built-in metrics to assess important quality metrics such as semantic relatedness and diversity [29].…”

Section: Supporting Evaluation and Experimentationmentioning

confidence: 99%

“…Training Conversational AI systems to deal with the expressiveness of natural language often requires collecting a large and linguistically diverse dataset of utterances [12,30]. Having experts to provide and annotate utterances at scale can be costly and time consuming, reasons that have motivated research into other utterance paraphrasing methods [29]. These approaches are generally grouped into those i) relying on deployed prototypes to collect utterances directly from users, ii) leveraging crowdsourcing to collect paraphrases at scale with non experts, and iii) automated approaches that generate paraphrases systematically.…”

Section: Introductionmentioning

confidence: 99%

An extensible and reusable pipeline for automated utterance paraphrases

et al. 2021

Self Cite

View full text Add to dashboard Cite

In this demonstration paper we showcase an extensible and reusable pipeline for automatic paraphrase generation , i.e., reformulating sentences using different words. Capturing the nuances of human language is fundamental to the effectiveness of Conversational AI systems, as it allows them to deal with the different ways users can utter their requests in natural language. Traditional approaches to utterance paraphrasing acquisition, such as hiring experts or crowd-sourcing, involve processes that are often costly or time consuming, and with their own trade-offs in terms of quality. Automatic paraphrasing is emerging as an attractive alternative that promises a fast, scalable and cost-effective process. In this paper we showcase how our extensible and reusable pipeline for automated utterance paraphrasing can support the development of Conversational AI systems by integrating and extending existing techniques under an unified and configurable framework.

show abstract

User Utterance Acquisition for Training Task-Oriented Bots: A Review of Challenges, Techniques and Opportunities

Cited by 13 publications

References 19 publications

Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness

Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness

API2CAN: a dataset & service for canonical utterance generation for REST APIs

An extensible and reusable pipeline for automated utterance paraphrases

Contact Info

Product

Resources

About