Asking Questions the Human Way: Scalable Question-Answer Generation from Text Corpus

Abstract

The ability to ask questions is important in both human and machine intelligence. Learning to ask questions helps knowledge acquisition, improves question-answering and machine reading comprehension tasks, and helps a chatbot to keep the conversation flowing with a human. Existing question generation models are ineffective at generating a large amount of high-quality question-answer pairs from unstructured text, since given an answer and an input passage, question generation is inherently a one-to-many mapping. In this paper, we propose Answer-Clue-Style-aware Question Generation (ACS-QG), which aims at automatically generating high-quality and diverse question-answer pairs from unlabeled text corpus at scale by imitating the way a human asks questions. Our system consists of: i) an information extractor, which samples from the text multiple types of assistive information to guide question generation; ii) neural question generators, which generate diverse and controllable questions, leveraging the extracted assistive information; and iii) a neural quality controller, which removes low-quality generated data based on text entailment. We compare our question generation models with existing approaches and resort to voluntary human evaluation to assess the quality of the generated question-answer pairs. The evaluation results suggest that our system dramatically outperforms state-of-the-art neural question generation models in terms of the generation quality, while being scalable in the meantime. With models trained on a relatively smaller amount of data, we can generate 2.8 million quality-assured question-answer pairs from a million sentences found in Wikipedia.

Evaluating BERT-based Rewards for Question Generation with Reinforcement Learning

“…As a dual task of question-answering, QG can be used to improve QA performance. Some works [7,11,26,46,58] take QG as a generator to harvest question-answer pairs from passages, and use this harvested data to pre-train QA models, which subsequently resulted in improved QA model effectiveness. QG is also widely used in IR tasks, such as improving search system effectiveness by generating clarifying questions [57], or generating questions from e-commercial customers reviews [55].…”

Improving the Robustness of QA Models to Challenge Sets with Variational Question-Answer Pair Generation

“…We used heuristics to filter out low-quality generated QA pairs, dropping questions that are longer than 20 words or shorter than 5 words and answers that are longer than 10 words, keeping questions that have at least one interrogative word, and removing n-gram repetition in questions. While some existing works used the BERT QA model or an entailment model as a data filter Zhang and Bansal, 2019;Liu et al, 2020), our heuristics are enough to obtain improvement in the downstream QA task as shown in §4.6. Some samples in our datasets are given in Table 3, showing that the diverse QA pairs are generated.…”

“…We also chose 100 samples from SQuAD Du test . In addition to the three items proposed by Liu et al (2020), we asked annotators if an given answer is important, i.e., it is worth being asked about. We showed the workers a triple (passage, question, answer) and asked them to answer the four questions shown in Table 4.…”

“…For the downstream QA task, most existing studies have evaluated QAG methods using a test set from the same distribution as a training set (Yang et al, 2017a;Zhang and Bansal, 2019;Liu et al, 2020). However, when a QA model is evaluated only on an in-distribution test set, it is difficult to verify that the model is not exploiting unintended biases in a dataset (Geirhos et al, 2020).…”

Sim2RealQA: Using Life Simulation to Solve Question Answering Real-World Events

“…As with the real-world QA, most existing works use the encoder-decoder framework that takes as input a question and a sequence of words in a given context and then generates answer words [55]- [57]. For augmenting the question and answer pairs to improve QA performance, the methods of generating questions as well as answers have also been proposed [58], [59]. However, these works mainly focus on a single domain for answer generation.…”