Bootstrapping Conversational Agents with Weak Supervision

Mallinar, Neil; Shah, Abhishek; Ugrani, Rajendra; Gupta, Ayush; Gurusankar, Manikandan; Ho, Tin Kam; Liao, Qi; Zhang, Yunfeng; Bellamy, Rachel K. E.; Yates, Robert; Desmarais, C.; McGregor, Blake

doi:10.1609/aaai.v33i01.33019528

Cited by 16 publications

(15 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Finally, our work improves dialogue quality by utilizing larger datasets with noisier labels than traditional supervision. Other applications of weak supervision to dialogue (Mallinar et al, 2019) and relation extraction have observed similar results (Bunescu and Mooney, 2007;Hancock et al, 2018;Ratner et al, 2017).…”

Section: Related Worksupporting

confidence: 61%

Learning from Dialogue after Deployment: Feed Yourself, Chatbot!

Hancock¹,

Bordes²,

Mazaré³

et al. 2019

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics

116

View full text Add to dashboard Cite

The majority of conversations a dialogue agent sees over its lifetime occur after it has already been trained and deployed, leaving a vast store of potential training signal untapped. In this work, we propose the self-feeding chatbot, a dialogue agent with the ability to extract new training examples from the conversations it participates in. As our agent engages in conversation, it also estimates user satisfaction in its responses. When the conversation appears to be going well, the user's responses become new training examples to imitate. When the agent believes it has made a mistake, it asks for feedback; learning to predict the feedback that will be given improves the chatbot's dialogue abilities further. On the PERSONACHAT chitchat dataset with over 131k training examples, we find that learning from dialogue with a selffeeding chatbot significantly improves performance, regardless of the amount of traditional supervision.

show abstract

Section: Related Worksupporting

confidence: 61%

Learning from Dialogue after Deployment: Feed Yourself, Chatbot!

Hancock¹,

Bordes²,

Mazaré³

et al. 2019

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics

116

View full text Add to dashboard Cite

show abstract

“…Recent works point to two directions to build quality intent models. Re-using available conversation log, to bootstrap intent model building process (Mallinar et al, 2018;Goyal et al, 2018;Shi et al, 2018;Liu et al, 2007). The other is to allow domain experts to build an intent model by working on the model definition, labeling, and evaluation through user interfaces (Williams et al, 2015).…”

Section: Related Workmentioning

confidence: 99%

“…Gathering good quality labeled data for any machine learning process is expensive. There have been significant efforts to reduce labeling effort; including work on clustering(Cheung and Li, 2012;Xu et al, 2017;Perkins and Yang, 2019), semi-supervised learning (Chapelle et al, 2010), active learning (Settles, 2012), transfer learning (Goyal et al, 2018) and also recently proposed data programming frameworks (Ratner et al, 2017;Mallinar et al, 2018). Semi-supervised, Transfer learning and active learning require seed training data for processing.…”

Section: Related Workmentioning

confidence: 99%

“…A transfer learning-based system has been proposed (Goyal et al, 2018) to learn from low resource settings. Data programming based (Mallinar et al, 2018) systems provide an interface to write labeling function for labeled data generation. However, one underlying assumption of using these methods is that they all require the intents to be known beforehand.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Intent Mining from past conversations for Conversational Agent

Chatterjee¹,

Sengupta²

2020

Proceedings of the 28th International Conference on Computational Linguistics

View full text Add to dashboard Cite

Conversational systems are of primary interest in the AI community. Organizations are increasingly using chatbot to provide round-the-clock support and to increase customer engagement. Many commercial bot building frameworks follow a standard approach that requires one to build and train an intent model to recognize user input. These frameworks require a collection of user utterances and corresponding intent to train an intent model. Collecting a substantial coverage of training data is a bottleneck in the bot building process. In cases where past conversation data is available, the cost of labeling hundreds of utterances with intent labels is time-consuming and laborious. In this paper, we present an intent discovery framework that can mine a vast amount of conversational logs and to generate labeled data sets for training intent models. We have introduced an extension to the DBSCAN (Ester et al., 1996) algorithm and presented a density-based clustering algorithm ITER-DBSCAN for unbalanced data clustering. Empirical evaluation on one conversation dataset, six intent dataset, and one short text clustering dataset show the effectiveness of our hypothesis. We release the datasets and code for future evaluation at https://github.com/ajaychatterjee/IntentMining . 1

show abstract

“…There has been a lot of work regarding semisupervision, for both image, video, and text classification (Li et al, 2019;Mallinar et al, 2019). Wang et al (2009), for example, applied semisupervised learning algorithms for video annotation.…”

Section: Related Workmentioning

confidence: 99%

How Many Users Are Enough? Exploring Semi-Supervision and Stylometric Features to Uncover a Russian Troll Farm

Nasrin¹,

Choo²,

Ko³

et al. 2019

Proceedings of the Second Workshop on Natural Language Processing for Internet Freedom: Censorship, Disinformation, and Propaga

View full text Add to dashboard Cite

Social media has reportedly been (ab)used by Russian troll farms to promote political agendas. Specifically, state-affiliated actors disguise themselves as native citizens of the United States to promote discord and promote their political motives. Therefore, developing methods to automatically detect Russian trolls can ensure fair elections and possibly reduce political extremism by stopping trolls that produce discord. While data exists for some troll organizations (e.g., Internet Research Agency), it is challenging to collect ground-truth accounts for new troll farms in a timely fashion. In this paper, we study the impact the number of labeled troll accounts has on detection performance. We analyze the use of self-supervision with less than 100 troll accounts as training data. We improve classification performance by nearly 4% F1. Furthermore, in combination with self-supervision, we also explore novel features for troll detection grounded in stylometry. Intuitively, we assume that the writing style is consistent across troll accounts because a single troll organization employee may control multiple user accounts. Overall, we improve on models based on words features by ∼9% F1.

show abstract

Bootstrapping Conversational Agents with Weak Supervision

Cited by 16 publications

References 8 publications

Learning from Dialogue after Deployment: Feed Yourself, Chatbot!

Learning from Dialogue after Deployment: Feed Yourself, Chatbot!

Intent Mining from past conversations for Conversational Agent

How Many Users Are Enough? Exploring Semi-Supervision and Stylometric Features to Uncover a Russian Troll Farm

Contact Info

Product

Resources

About