Retrieval Enhanced Model for Commonsense Generation

Abstract

Commonsense generation is a challenging task of generating a plausible sentence describing an everyday scenario using provided concepts. Its requirement of reasoning over commonsense knowledge and compositional generalization ability even puzzles strong pre-trained language generation models. We propose a novel framework using retrieval methods to enhance both the pre-training and fine-tuning for commonsense generation. We retrieve prototype sentence candidates by concept matching and use them as auxiliary input. For finetuning, we further boost its performance with a trainable sentence retriever. We demonstrate experimentally on the large-scale Common-Gen benchmark that our approach achieves new state-of-the-art results. 1 * Work done during internship at Microsoft. 1 The code and data are available at https://github. com/HanNight/RE-T5 Concept Set #1: dog, frisbee, catch, throw Gold Target Sentences: A dog leaps to catch a thrown frisbee. The dog catches the frisbee when the boy throws it. A man throws away his dog 's favorite frisbee expecting him to catch it in the air. Concept Set #2: lake, shore, canoe Gold Target Sentences: Canoe on a shore of lake. Canoe on shore with rainbow across the lake. Several canoes parked in the grass on the shore of a lake.

KGR4: Retrieval, Retrospect, Refine and Rethink for Commonsense Generation

“…To investigate the effectiveness of our framework, we conduct extensive experiments on the CommonGen benchmark, where experimental results and in-depth analysis demon-strate the superiority of our framework. Specifically, KGR 4 significantly surpasses the previous best result (Wang et al 2021)…”

“…Therefore, the performance of this line of work would be limited. More recently, Fan et al (2020) and Wang et al (2021) introduce retrievers to search auxiliary information from external plain sentences, which contains enormous daily scenarios. Intuitively, the implicit commonsense knowledge within plain sentences is more abundant than that in human-annotated knowledge bases.…”

“…Typically, Fan et al (2020) equip the BART-based generation model with an enhanced knowledge injection module, which exploits the commonsense knowledge extracted from retrieved sentences. Furthermore, Wang et al (2021) propose a trainable retriever to search auxiliary sentences for given concepts, and concatenate them with concepts as input for T5 (Raffel et al 2020a). Obviously, our framework is an extension of retrievaland-generation methods.…”

“…We firstly use the concept mapping (Wang et al 2021) and roughly search candidate prototypes for the concept set x from an external corpus D e , which consists of caption sentences describing the daily scenario. Then, we train a RoBERTa-based binary classifier as a scorer to semantically evaluate candidate prototypes.…”

Revisiting Generative Commonsense Reasoning: A Pre-Ordering Approach

“…There are two major approaches to enhance the vanilla PTM's ability of commonsense reasoning on generation. The first approach is to introduce explicit knowledge from external sources such as ConceptNet and retrieved prototypes (Fan et al, 2020;Wang et al, 2021), which can facilitate GSR by either building connections between related concepts or providing adjunct words for the input. The second approach is to explicitly teach models to reason over the concepts via new pre-training objectives (Zhou et al, 2021).…”