Few-Shot NLG with Pre-Trained Language Model

Abstract

Neural-based end-to-end approaches to natural language generation (NLG) from structured data or knowledge are data-hungry, making their adoption for real-world applications difficult with limited data. In this work, we propose the new task of few-shot natural language generation. Motivated by how humans tend to summarize tabular data, we propose a simple yet effective approach and show that it not only demonstrates strong performance but also provides good generalization across domains. The design of the model architecture is based on two aspects: content selection from input data and language modeling to compose coherent sentences, which can be acquired from prior knowledge. With just 200 training examples, across multiple domains, we show that our approach achieves very reasonable performances and outperforms the strongest baseline by an average of over 8.0 BLEU points improvement. Our code and data can be found at https:

Logic2Text: High-Fidelity Natural Language Generation from Logical Forms

“…Most recently, large-scale pre-trained models (Radford et al, 2019;Song et al, 2019;Raffel et al, 2019) have achieved new state-ofthe-arts on various generation tasks. Chen et al (2019b) demonstrate that a simple pre-training based method can achieve very reasonable performance on the WikiBio dataset (Lebret et al, 2016) under few-shot setting. More recent works begin to focus on fidelity preserving of the generation, such as (Dhingra et al, 2019;Tian et al, 2019).…”

“…Considering that acquiring a large amount of (logical form, description) pairs in real-world cases is expensive, we also include a few-shot learning task for our dataset, where the model is only provided with hundreds of paired examples. Previous works have shown that the pre-trained language models obtain strong NLG performance even with a handful of fine-tuning instances (Chen et al, 2019b). Therefore we still use the best-performing GPT-2 model for this study.…”

“…Natural language generation (NLG) from structured data has been an important research problem in many applications. Recent data-driven methods have achieved good performances on various NLG tasks (Liu et al, 2018;Freitag and Roy, 2018;Chen et al, 2019b). However most studies focus on surface descriptions of simple record sequences, for example, attribute-value pairs of fixed or very limited schema, like E2E (Novikova et al, 2017) and WikiBio (Lebret et al, 2016).…”

Best Practices for Data-Efficient Modeling in NLG:How to Train Production-Ready Neural Models with Less Data

“…In similar studies, pretrained GPT models (Radford et al, 2019) were used by Chen et al (2020) and Peng et al (2020), who fine-tune them on a small set of in-domain data, but they did not distill these models into ones suitable for production. Interestingly, Wen et al (2016) demonstrated that the structure of arguments in existing dialogues can be used to guide data collection for low-resource domain adaptation, which is similar to the bucketing strategies explored here.…”

“…More recently, advances in neural-network-based (conditional) language generation prompted a new direction in NLG research (Novikova et al, 2017;Budzianowski et al, 2018;Chen et al, 2020;Bal-akrishnan et al, 2019;Peng et al, 2020). The process is typically split into two steps: (1) serialization of input data into a flattened meaning representation (MR), and (2) using the neural generation model to generate a natural language response conditioned on the MR.…”