Pretraining Methods for Dialog Context Representation Learning

Mehri, Shikib; Razumovskaia, Evgeniia; Zhao, Tiancheng; Eskénazi, Maxine

doi:10.18653/v1/p19-1373

Cited by 63 publications

(72 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…When it comes to dialog systems, response selection provides a more suitable pretraining task for learning representations that can encapsulate conversational cues. Such models can be pretrained using large corpora of natural unlabelled conversational data (Henderson et al, 2019b;Mehri et al, 2019). Response selection is also directly applicable to retrieval-based dialog systems, a popular and elegant approach to framing dialog (Wu et al, 2017;Weston et al, 2018;Mazaré et al, 2018;Gunasekara et al, 2019;Henderson et al, 2019b).…”

Section: Introductionmentioning

confidence: 99%

ConveRT: Efficient and Accurate Conversational Representations from Transformers

Henderson¹,

Casanueva²,

Mrkšić³

et al. 2020

Findings of the Association for Computational Linguistics: EMNLP 2020

105

View full text Add to dashboard Cite

General-purpose pretrained sentence encoders such as BERT are not ideal for real-world conversational AI applications; they are computationally heavy, slow, and expensive to train. We propose ConveRT (Conversational Representations from Transformers), a pretraining framework for conversational tasks satisfying all the following requirements: it is effective, affordable, and quick to train. We pretrain using a retrieval-based response selection task, effectively leveraging quantization and subword-level parameterization in the dual encoder to build a lightweight memoryand energy-efficient model. We show that Con-veRT achieves state-of-the-art performance across widely established response selection tasks. We also demonstrate that the use of extended dialog history as context yields further performance gains. Finally, we show that pretrained representations from the proposed encoder can be transferred to the intent classification task, yielding strong results across three diverse data sets. ConveRT trains substantially faster than standard sentence encoders or previous state-of-the-art dual encoders. With its reduced size and superior performance, we believe this model promises wider portability and scalability for Conversational AI applications.

show abstract

Section: Introductionmentioning

confidence: 99%

ConveRT: Efficient and Accurate Conversational Representations from Transformers

Henderson¹,

Casanueva²,

Mrkšić³

et al. 2020

Findings of the Association for Computational Linguistics: EMNLP 2020

105

View full text Add to dashboard Cite

show abstract

“…This idea can also be applied to task-oriented dialog systems to transfer general natural language knowledge from large-scale corpora to a specific dialog task. Some early studies have shown the possibility of using pre-training models to model task-oriented dialogs [46,99,100,130,131].…”

Section: Discussion and Future Trendsmentioning

confidence: 99%

Recent advances and challenges in task-oriented dialog systems

Zhang

Takanobu

Huang

et al. 2020

Sci. China Technol. Sci.

105

View full text Add to dashboard Cite

Due to the significance and value in human-computer interaction and natural language processing, task-oriented dialog systems are attracting more and more attention in both academic and industrial communities. In this paper, we survey recent advances and challenges in task-oriented dialog systems. We also discuss three critical topics for task-oriented dialog systems: (1) improving data efficiency to facilitate dialog modeling in low-resource settings, (2) modeling multi-turn dynamics for dialog policy learning to achieve better task-completion performance, and (3) integrating domain ontology knowledge into the dialog model. Besides, we review the recent progresses in dialog evaluation and some widely-used corpora. We believe that this survey, though incomplete, can shed a light on future research in task-oriented dialog systems. task-oriented dialog systems, natural language understanding, dialog policy, dialog state tracking, natural language generation

show abstract

“…In recent years, the pretraining models have a huge impact in the field of natural language understanding and natural language generation. Mehri et al [58] studied the sentence representation based on the pretraining models and employed the pretrained representation in dialogue generation task. They proved that the pretraining model can be utilized in open-dialogue generation task.…”

Section: Pretraining-model-based Methodsmentioning

confidence: 99%

Neural Dialogue Generation Methods in Open Domain: A Survey

Sun¹,

Li²

2021

NLPRE

View full text Add to dashboard Cite

Open-Domain Dialogue Generation (human-computer interaction) is an important issue in the field of Natural Language Processing (NLP). Because of the improvement of deep learning techniques, a large number of neural dialogue generative methods were proposed to generate better responses. In this survey, we elaborated the research history of these existing generative methods, and then roughly divided them into six categories, i.e., Encoder-Decoder framework-based methods, Hierarchical Recurrent Encoder-Decoder (HRED)-based methods, Variational Autoencoder (VAE)-based methods, Reinforcement Learning (RL)-based methods, Generative Adversarial Network (GAN)-based methods, and pretraining-model-based methods. We dived into the methods of each category and gave the detailed discussions of these methods. After that, we presented a comparison among the different categories of methods and analyzed their advantages and disadvantages. We enumerated some open access public datasets and some commonly used automatic evaluating metrics. Finally, we discuss some possible research directions that can take the research of neural dialogue generation into a new frontier in the future.

show abstract

Pretraining Methods for Dialog Context Representation Learning

Cited by 63 publications

References 27 publications

ConveRT: Efficient and Accurate Conversational Representations from Transformers

ConveRT: Efficient and Accurate Conversational Representations from Transformers

Recent advances and challenges in task-oriented dialog systems

Neural Dialogue Generation Methods in Open Domain: A Survey

Contact Info

Product

Resources

About