Training Question Answering Models From Synthetic Data

Abstract

Question and answer generation is a data augmentation method that aims to improve question answering (QA) models given the limited amount of human labeled data. However, a considerable gap remains between synthetic and human-generated question-answer pairs. This work aims to narrow this gap by taking advantage of large language models and explores several factors such as model size, quality of pretrained models, scale of data synthesized, and algorithmic choices. On the SQUAD1.1 question answering task, we achieve higher accuracy using solely synthetic questions and answers than when using the SQUAD1.1 training set questions alone. Removing access to real Wikipedia data, we synthesize questions and answers from a synthetic text corpus generated by an 8.3 billion parameter GPT-2 model and achieve 88.4 Exact Match (EM) and 93.9 F1 score on the SQUAD1.1 dev set. We further apply our methodology to SQUAD2.0 and show a 2.8 absolute gain on EM score compared to prior work using synthetic data.

End-to-End Synthetic Data Generation for Domain Adaptation of Question Answering Systems

“…SQuAD 1.1 dev set is used to select the best model during training. As a baseline for QA data generation, we implemented a three-stage pipeline similar to the state-of-the-art approach of Puri et al (2020). We call this baseline QGen, which generates a question given a passage and extracted span, q ∼ p(q|a, c).…”

“…Recent work have used the round-trip filtering method Puri et al, 2020) to prune the synthetic QA set and improve data quality. This method consists of two steps: (1) using an RC model to provide answers to the generated questions; (2) dropping the QA pairs for which the answer of the RC model does not match the span detected answer.…”

“…However, they only got improvement when only a small amount of human labeled data is available. Recently, with the help of large pre-trained language models, Alberti et al ( 2019) and Puri et al (2020) have been able to improve the performance of RC models using generated questions. However, they need two extra BERT models to identify high-quality answer spans, and filter out low-quality questionanswer pairs.…”

“…Although many past works have proposed different strategies for question generation, they have limited or no success in improving the downstream QA task (Du et al, 2017;Sun et al, 2018;Song et al, 2018;Klein and Nabi, 2019;Wang et al, 2020;Ma et al, 2020;Chen et al, 2020;Tuan et al, 2019). Some recent approaches for synthetic QA data generation based on large pretrained language models (LM) have started to demonstrate success in improving the downstream Reading Comprehension (RC) task with automatically generated data Puri et al, 2020). However, these approaches typically consist of multi-stage systems that use three modules: span/answer detector, question generator and question filtering.…”

Generative Data Augmentation for Commonsense Reasoning

“…These approaches use generative adversarial networks (Zhao et al, 2018b) and populationbased optimization algorithms (Alzantot et al, 2018). Previous work has also presented methods to generate questions for reading comprehension (Heilman and Smith, 2010;Rus et al, 2011;Alberti et al, 2019;Puri et al, 2020a), online tutoring (Lindberg et al, 2013), factual QA (Serban et al, 2016) and visual question generation (Mostafazadeh et al, 2016). A comprehensive survey on neural question generation can be found in Pan et al (2019).…”