Retrieval Enhanced Model for Commonsense Generation

Abstract

Commonsense generation is a challenging task of generating a plausible sentence describing an everyday scenario using provided concepts. Its requirement of reasoning over commonsense knowledge and compositional generalization ability even puzzles strong pre-trained language generation models. We propose a novel framework using retrieval methods to enhance both the pre-training and fine-tuning for commonsense generation. We retrieve prototype sentence candidates by concept matching and use them as auxiliary input. For finetuning, we further boost its performance with a trainable sentence retriever. We demonstrate experimentally on the large-scale Common-Gen benchmark that our approach achieves new state-of-the-art results. 1 * Work done during internship at Microsoft. 1 The code and data are available at https://github. com/HanNight/RE-T5 Concept Set #1: dog, frisbee, catch, throw Gold Target Sentences: A dog leaps to catch a thrown frisbee. The dog catches the frisbee when the boy throws it. A man throws away his dog 's favorite frisbee expecting him to catch it in the air. Concept Set #2: lake, shore, canoe Gold Target Sentences: Canoe on a shore of lake. Canoe on shore with rainbow across the lake. Several canoes parked in the grass on the shore of a lake.

Contextualized Scene Imagination for Generative Commonsense Reasoning

“…Baselines (1) Concept2Sentence: We consider several recent submissions to the leaderboard of CommonGen that leverage auxiliary information for GCSR. KFCNet (Li et al, 2021), Re-T5 (Wang et al, 2021), and EKI-BART (Fan et al, 2020) are prototype-based models, which retrieve sentences containing as many input concepts as possible from external captions and NLI datasets, and then use these sentences as auxiliary inputs. VisCTG (Feng et al, 2021b) is an image-augmented model which retrieves images from Google by using concepts as a query, followed by an image captioning model that generates captions as auxiliary inputs.…”

“…This is because LMs have no intrinsic mechanism to reason over high-level relations between concepts . To close the knowledge gap, recent work augment LM input with knowledge graph triples (e.g., (dog, CapableOf, catch)) retrieved from ConceptNet Li et al, 2020), or prototype sentences that cover input concepts retrieved from external text corpora (Fan et al, 2020;Wang et al, 2021). However, despite the input augmentation, GCSR skills are implicitly learned based on the concept-text pairs in the training data, without explicit supervision.…”

“…Knowledge-Enhanced GCSR Recent works Li et al, 2021) on GCSR propose to retrieve external knowledge to enhance the text generation. Prototype-based models, including EKI-BART (Fan et al, 2020), Re-T5 (Wang et al, 2021), and KFCNet (Li et al, 2021) retrieve massive prototype sentences from external corpora (over 70M) like visual captions and Wikipedia as auxiliary input to the LM. Though the retrieved prototype sentences provide high coverage on the concepts, their model is supervised to compose sentences that are very similar to those existing prototypes.…”