A Controllable Model of Grounded Response Generation

Wu, Zeqiu; Galley, Michel; Brockett, Chris; Zhang, Yizhe; Gao, Xiang; Quirk, Chris; Koncel-Kedziorski, Rik; Gao, Jianfeng; Hajishirzi, Hannaneh; Ostendorf, Mari; Dolan, Bill

doi:10.48550/arxiv.2005.00613

Cited by 13 publications

(13 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The credibility of a text contains multiple aspects such as coherency, clarity, veracity, etc. For non-pretraining methods, previous works explored many approaches to promote several aspects of credible generation: Plan-and-write [35] uses a hierarchical method that first plans a storyline and then generates a story based on it to improve the text coherency; CGRG [36] uses grounding knowledge to generate veritable answers. With the development of pre-training technology, large-scale language models provide a simple yet powerful solution for credible text generation.…”

Section: Credible Text Generationmentioning

confidence: 99%

ERNIE 3.0 Titan: Exploring Larger-scale Knowledge Enhanced Pre-training for Language Understanding and Generation

Wang¹,

Sun²,

Xiang³

et al. 2021

Preprint

View full text Add to dashboard Cite

Pre-trained language models have achieved state-of-the-art results in various Natural Language Processing (NLP) tasks. has shown that scaling up pre-trained language models can further exploit their enormous potential. A unified framework named ERNIE 3.0 [2] was recently proposed for pre-training large-scale knowledge enhanced models and trained a model with 10 billion parameters. ERNIE 3.0 outperformed the state-of-the-art models on various NLP tasks. In order to explore the performance of scaling up ERNIE 3.0, we train a hundred-billion-parameter model called ERNIE 3.0 Titan with up to 260 billion parameters on the PaddlePaddle [3] platform. Furthermore, we design a self-supervised adversarial loss and a controllable language modeling loss to make ERNIE 3.0 Titan generate credible and controllable texts. To reduce the computation overhead and carbon emission, we propose an online distillation framework for ERNIE 3.0 Titan, where the teacher model will teach students and train itself simultaneously. ERNIE 3.0 Titan is the largest Chinese dense pre-trained model so far. Empirical results show that the ERNIE 3.0 Titan outperforms the state-of-the-art models on 68 NLP datasets.

show abstract

Section: Credible Text Generationmentioning

confidence: 99%

ERNIE 3.0 Titan: Exploring Larger-scale Knowledge Enhanced Pre-training for Language Understanding and Generation

Wang¹,

Sun²,

Xiang³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…The skeleton could also be multiple keyphrases. The keyphrases are extracted based on word frequency (Ippolito et al, 2019;Tan et al, 2020;Wu et al, 2020), an off-the-shelf keyword extraction method (Peng et al, 2018;Goldfarb-Tarrant et al, 2019;Yao et al, 2019;Rashkin et al, 2020;Zhang et al, 2020), a sentence compression dataset and reinforcement learning (Xu et al, 2018), or image caption datasets and ConceptNet (Lin et al, 2020). Most of the studies focus on modeling the longterm dependency among the keyphrases and/or forcing the generation to contain the keyphrases.…”

Section: Related Workmentioning

confidence: 99%

Changing the Mind of Transformers for Topically-Controllable Language Generation

Chang¹,

Yuan²,

Iyyer³

et al. 2021

Preprint

View full text Add to dashboard Cite

Large Transformer-based language models can aid human authors by suggesting plausible continuations of text written so far. However, current interactive writing assistants do not allow authors to guide text generation in desired topical directions. To address this limitation, we design a framework that displays multiple candidate upcoming topics, of which a user can select a subset to guide the generation. Our framework consists of two components: (1) a method that produces a set of candidate topics by predicting the centers of word clusters in the possible continuations, and ( 2) a text generation model whose output adheres to the chosen topics. The training of both components is self-supervised, using only unlabeled text. Our experiments demonstrate that our topic options are better than those of standard clustering approaches, and our framework often generates fluent sentences related to the chosen topics, as judged by automated metrics and crowdsourced workers.

show abstract

“…Researchers have also explored the possibility of using other latent semantic forms such as topic (Wang et al 2017), sentence function (Ke et al 2018;Bi et al 2019), frame semantics (Gupta et al 2020), lexical phrases (Wu et al 2020), or goals in conversation (Tang et al 2019). Previous works have mainly focused on fusing the information of much simpler semantic forms into generation, thus altering few words in the returned response and might not be as effective.…”

Section: Utilizing Latent Semantics Formsmentioning

confidence: 99%

Predict and Use Latent Patterns for Short-Text Conversation

Chen,

Chao,

Chao

et al. 2020

Preprint

View full text Add to dashboard Cite

Many neural network models nowadays have achieved promising performances in Chit-chat settings. The majority of them rely on an encoder for understanding the post and a decoder for generating the response. Without given assigned semantics, the models lack the fine-grained control over responses as the semantic mapping between posts and responses is hidden on the fly within the end-to-end manners. Some previous works utilize sampled latent words as a controllable semantic form to drive the generated response around the work, but few works attempt to use more complex semantic forms to guide the generation. In this paper, we propose to use more detailed semantic forms, including latent responses and part-of-speech sequences sampled from the corresponding distributions, as the controllable semantics to guide the generation. Our experimental results show that the richer semantics are not only able to provide informative and diverse responses, but also increase the overall performance of response quality, including fluency and coherence.

show abstract

A Controllable Model of Grounded Response Generation

Cited by 13 publications

References 18 publications

ERNIE 3.0 Titan: Exploring Larger-scale Knowledge Enhanced Pre-training for Language Understanding and Generation

ERNIE 3.0 Titan: Exploring Larger-scale Knowledge Enhanced Pre-training for Language Understanding and Generation

Changing the Mind of Transformers for Topically-Controllable Language Generation

Predict and Use Latent Patterns for Short-Text Conversation

Contact Info

Product

Resources

About