KFCNet: Knowledge Filtering and Contrastive Learning for Generative Commonsense Reasoning

Abstract

Pre-trained language models have led to substantial gains over a broad range of natural language processing (NLP) tasks, but have been shown to have limitations for natural language generation tasks with high-quality requirements on the output, such as commonsense generation and ad keyword generation. In this work, we present a novel Knowledge Filtering and Contrastive learning Network (KFCNet) which references external knowledge and achieves better generation performance. Specifically, we propose a BERTbased filter model to remove low-quality candidates, and apply contrastive learning separately to each of the encoder and decoder, within a general encoder-decoder architecture. The encoder contrastive module helps to capture global target semantics during encoding, and the decoder contrastive module enhances the utility of retrieved prototypes while learning general features. Extensive experiments on the CommonGen benchmark show that our model outperforms the previous state of the art by a large margin: +6.6 points (42.5 vs. 35.9) for +3.7 points (33.3 vs. 29.6) for SPICE, and +1.3 points (18.3 vs. 17.0) for CIDEr. We further verify the effectiveness of the proposed contrastive module on ad keyword generation, and show that our model has potential commercial value.

A Dog Is Passing Over The Jet? A Text-Generation Dataset for Korean Commonsense Reasoning and Evaluation

“…RE-T5 (Wang et al, 2021) reinforces the input value by using a retriever to import sentences related to the concepts from external knowledge. KFCNet (Li et al, 2021) presents the state-ofthe-art performance in CommonGen by removing low-quality sentences in external knowledge and applying contrastive learning, respectively. However, there does not exist Korean dataset for generative commonsense reasoning and advanced research as well.…”

Bridging the Gap between Pre-Training and Fine-Tuning for Commonsense Generation

“…Existing methods employ the Pre-trained Languege Models (PLMs) such as BART (Lewis et al, 2020), GPT-2 (Radford et al, 2019) as the backbone to solve this problem. They Fan et al, 2020;Wang et al, 2021;Li et al, 2021) usually take the concatenated concepts words as the inputs. However, such processing of inputs * This work was done during an internship at Tencent.…”