A Knowledge-Enhanced Pretraining Model for Commonsense Story Generation

Abstract

Story generation, namely, generating a reasonable story from a leading context, is an important but challenging task. In spite of the success in modeling fluency and local coherence, existing neural language generation models (e.g., GPT-2) still suffer from repetition, logic conflicts, and lack of long-range coherence in generated stories. We conjecture that this is because of the difficulty of associating relevant commonsense knowledge, understanding the causal relationships, and planning entities and events with proper temporal order. In this paper, we devise a knowledge-enhanced pretraining model for commonsense story generation. We propose to utilize commonsense knowledge from external knowledge bases to generate reasonable stories. To further capture the causal and temporal dependencies between the sentences in a reasonable story, we use multi-task learning, which combines a discriminative objective to distinguish true and fake stories during fine-tuning. Automatic and manual evaluation shows that our model can generate more reasonable stories than state-of-the-art baselines, particularly in terms of logic and global coherence.

Robo-writers: the rise and risks of language-generating AI

“…One future path lies in combining language models with knowledge bases: curated databases of declarative facts. In work presented at last year's Association for Computational Linguistics meeting 9 , researchers fine-tuned GPT-2 on sentences explicitly stating facts and inferences from a compendium of common sense (for instance, if someone cooks spaghetti, that person wants to eat). As a result, it wrote short stories that were more logical.…”

Language Generation with Multi-Hop Reasoning on Commonsense Knowledge Graph

“…To address this defect, incorporating external commonsense knowledge to enhance models' reasoning ability has been widely explored (Lin et al, 2019;Ye et al, 2019;Lv et al, 2019). In language generation, previous work (Bhagavatula et al, 2020;Guan et al, 2020) transfers commonsense knowledge into pre-trained language models by utilizing triple information in commonsense knowledge bases such as ConceptNet (Speer and Havasi, 2012) and ATOMIC .…”

“…First, recovering knowledge triples at the posttraining stage (Guan et al, 2020) hardly enables the model to utilize the encoded knowledge in fine-tuning generation tasks which requires reasoning over underlying commonsense knowledge. Second, it ignores the abundant structural relational relevance of the concepts in the knowledge graph (Guan et al, 2020;Bhagavatula et al, 2020) that may provide multiple plausible evidence for complex reasoning.…”

“…First, recovering knowledge triples at the posttraining stage (Guan et al, 2020) hardly enables the model to utilize the encoded knowledge in fine-tuning generation tasks which requires reasoning over underlying commonsense knowledge. Second, it ignores the abundant structural relational relevance of the concepts in the knowledge graph (Guan et al, 2020;Bhagavatula et al, 2020) that may provide multiple plausible evidence for complex reasoning. Thus a richer and more explicit way of utilizing external commonsense knowledge is to exploit both structural and semantic information of the knowledge graph and reason over multihop relational paths where multiple connected triples provide chains of evidence for grounded text generation.…”

“…Recently, some work also attempted to integrate external commonsense knowledge into generative pretrained language models such as GPT-2 (Radford et al, 2019). Guan et al (2020) conducted posttraining on sythetic data constructed from commonsense knowledge bases by translating triplets into natural language texts using templates. Bhagavatula et al (2020) transferred embeddings of COMeT (Bosselut et al, 2019), a GPT-2 model fine-tuned to generate the tail entity of a triple in commonsense knowledge graph, into another GPT-2 model for text generation.…”