A Recurrent BERT-based Model for Question Generation

Chan, Ying-Hong; Fan, Yao-Chung

doi:10.18653/v1/d19-5821

Cited by 122 publications

(72 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Reinforcement learning is a popular approach to train the neural QG models, where the reward is defined as the evaluation metrics (Song et al, 2017;Kumar et al, 2018), or the QA accuracy/likelihood (Yuan et al, 2017;Hosking and Riedel, 2019;Zhang and Bansal, 2019). State-ofthe-art QG models Chan and Fan, 2019) use pre-trained language models. Question-Answer Pair Generation (QAG) from contexts, which is our main target, is a relatively less explored topic tackled by only a few recent works (Du and Cardie, 2018;.…”

Section: Related Workmentioning

confidence: 99%

Generating Diverse and Consistent QA pairs from Contexts with Information-Maximizing Hierarchical Conditional VAEs

Lee¹,

Lee²,

Jeong³

et al. 2020

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics

View full text Add to dashboard Cite

One of the most crucial challenges in question answering (QA) is the scarcity of labeled data, since it is costly to obtain question-answer (QA) pairs for a target text domain with human annotation. An alternative approach to tackle the problem is to use automatically generated QA pairs from either the problem context or from large amount of unstructured texts (e.g. Wikipedia). In this work, we propose a hierarchical conditional variational autoencoder (HCVAE) for generating QA pairs given unstructured texts as contexts, while maximizing the mutual information between generated QA pairs to ensure their consistency. We validate our Information Maximizing Hierarchical Conditional Variational AutoEncoder (Info-HCVAE) on several benchmark datasets by evaluating the performance of the QA model (BERT-base) using only the generated QA pairs (QA-based evaluation) or by using both the generated and human-labeled pairs (semisupervised learning) for training, against stateof-the-art baseline models. The results show that our model obtains impressive performance gains over all baselines on both tasks, using only a fraction of data for training. 1

show abstract

Section: Related Workmentioning

confidence: 99%

Generating Diverse and Consistent QA pairs from Contexts with Information-Maximizing Hierarchical Conditional VAEs

Lee¹,

Lee²,

Jeong³

et al. 2020

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics

View full text Add to dashboard Cite

show abstract

“…However, only few attempts have been made so far to make use of these pre-trained models for conditional language modeling. Dong et al (2019) and Chan and Fan (2019) use a single BERT model for both encoding and decoding and achieve state-of-the-art results in QG. However, both of them use the [MASK] token as the input for predicting the word in place, which makes the training slower as it warranties recurrent generation (Chan and Fan, 2019) or generation with random masking (Dong et al, 2019).…”

Section: Related Workmentioning

confidence: 99%

“…For evaluating our models, we report standard metrics of BLEU4, METEOR and ROUGE-L. As baselines, we take two of the non-BERT state-of-the-art models (Du and Cardie, 2018;Zhang and Bansal, Model BLEU4 METEOR ROUGE-L CorefNQG (Du and Cardie, 2018) 15.16 19.12 -SemdriftQG (Zhang and Bansal, 2019) 18.37 22.65 6.68 Recurrent-BERT (Chan and Fan, 2019) 20.33 23.88 48.23 UniLM (Dong et al, 2019) 22 Du et al (2017). BERT refers to BERT-Large(cased) model (Devlin et al, 2019) 2019) and the two BERT-based QG models (Dong et al, 2019;Chan and Fan, 2019). We experimented with 4 settings: one without using any copy mechanism (No Copy), one using normal copy (Normal Copy; §3.3.1), one using self-copy (Self-Copy; §3.3.2) and finally with two-hop selfcopy (Two-Hop Self-Copy; §3.3.3).…”

Section: Evaluation Metrics and Modelsmentioning

confidence: 99%

“…It took CopyBERT around 14 hours on a single GPU with 12GB main memory to train for 3 epochs, whereas UniLM took around 45 hours on the same hardware to train for 10 epochs to achieve similar results as reported in Dong et al (2019). We expect Recurrent-BERT (Chan and Fan, 2019) to take even longer time to train due to its sequential nature.…”

Section: Training Speedmentioning

confidence: 99%

See 1 more Smart Citation

CopyBERT: A Unified Approach to Question Generation with Self-Attention

Varanasi¹,

Amin²,

Neumann³

2020

Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI

View full text Add to dashboard Cite

Contextualized word embeddings provide better initialization for neural networks that deal with various natural language understanding (NLU) tasks including question answering (QA) and more recently, question generation (QG). Apart from providing meaningful word representations, pre-trained transformer models, such as BERT also provide self-attentions which encode syntactic information that can be probed for dependency parsing and POStagging. In this paper, we show that the information from self-attentions of BERT are useful for language modeling of questions conditioned on paragraph and answer phrases. To control the attention span, we use semidiagonal mask and utilize a shared model for encoding and decoding, unlike sequence-tosequence. We further employ copy mechanism over self-attentions to achieve state-of-the-art results for question generation on SQuAD dataset.

show abstract

“…One of the benefits of our architecture is that the modules are not bounded by any specific model. For the current work, we employ the QG model proposed by Chan and Fan (2019) for the two assistants of the teacher and for the student, taking advantage of BERT .…”

Section: Question Generation Modulementioning

confidence: 99%

Regularization of Distinct Strategies for Unsupervised Question Generation

Kang¹,

Hong²,

Roman³

et al. 2020

Findings of the Association for Computational Linguistics: EMNLP 2020

View full text Add to dashboard Cite

Unsupervised question answering (UQA) has been proposed to avoid the high cost of creating high-quality datasets for QA. One approach to UQA is to train a QA model with questions generated automatically. However, the generated questions are either too similar to a word sequence in the context or too drifted from the semantics of the context, thereby making it difficult to train a robust QA model. We propose a novel regularization method based on teacher-student architecture to avoid bias toward a particular question generation strategy and modulate the process of generating individual words when a question is generated. Our experiments demonstrate that we have achieved the goal of generating higher-quality questions for UQA across diverse QA datasets and tasks. We also show that this method can be useful for creating a QA model with few-shot learning.

show abstract

A Recurrent BERT-based Model for Question Generation

Cited by 122 publications

References 17 publications

Generating Diverse and Consistent QA pairs from Contexts with Information-Maximizing Hierarchical Conditional VAEs

Generating Diverse and Consistent QA pairs from Contexts with Information-Maximizing Hierarchical Conditional VAEs

CopyBERT: A Unified Approach to Question Generation with Self-Attention

Regularization of Distinct Strategies for Unsupervised Question Generation

Contact Info

Product

Resources

About