Simple and Effective Semi-Supervised Question Answering

Dhingra, Bhuwan; Danish, Danish; Rajagopal, Dheeraj

doi:10.18653/v1/n18-2092

Cited by 66 publications

(75 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Table 1 shows the result in context of published baselines and supervised results. Our approach significantly outperforms baseline systems and Dhingra et al (2018) and surpasses early supervised methods.…”

Section: Em F1mentioning

confidence: 86%

“…As we cannot assume access to a development dataset when training unsupervised models, the QA model training is halted when QA performance on a held-out set of synthetic QA data plateaus. We do, however, use the SQuAD development set to assess which model components are (Dhingra et al, 2018) 3.2 † 6.8 † BiDAF+SA (Dhingra et al, 2018) ‡ 10.0* 15.0* BERT-Large (Dhingra et al, 2018) ‡ 28.4* 35.8*…”

Section: Unsupervised Qa Experimentsmentioning

confidence: 99%

“…* indicates results on SQuAD dev set. † indicates results on non-standard test set created byDhingra et al (2018). ‡ indicates our re-implementation important (Section 3.2).…”

mentioning

confidence: 99%

See 2 more Smart Citations

Unsupervised Question Answering by Cloze Translation

Lewis¹,

Denoyer²,

Riedel³

2019

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics

111

101

View full text Add to dashboard Cite

Obtaining training data for Question Answering (QA) is time-consuming and resourceintensive, and existing QA datasets are only available for limited domains and languages. In this work, we explore to what extent high quality training data is actually required for Extractive QA, and investigate the possibility of unsupervised Extractive QA. We approach this problem by first learning to generate context, question and answer triples in an unsupervised manner, which we then use to synthesize Extractive QA training data automatically. To generate such triples, we first sample random context paragraphs from a large corpus of documents and then random noun phrases or named entity mentions from these paragraphs as answers. Next we convert answers in context to "fill-in-the-blank" cloze questions and finally translate them into natural questions. We propose and compare various unsupervised ways to perform cloze-tonatural question translation, including training an unsupervised NMT model using nonaligned corpora of natural questions and cloze questions as well as a rule-based approach. We find that modern QA models can learn to answer human questions surprisingly well using only synthetic training data. We demonstrate that, without using the SQuAD training data at all, our approach achieves 56.4 F1 on SQuAD v1 (64.5 F1 when the answer is a Named entity mention), outperforming early supervised models.

show abstract

Section: Em F1mentioning

confidence: 86%

Section: Unsupervised Qa Experimentsmentioning

confidence: 99%

See 1 more Smart Citation

Unsupervised Question Answering by Cloze Translation

Lewis¹,

Denoyer²,

Riedel³

2019

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics

111

101

View full text Add to dashboard Cite

show abstract

“…There are limited studies on semi-supervised learning on RC tasks Dhingra et al, 2018). In this section, we explore this possibility with virtual adversarial training.…”

Section: Is Semi-supervised Learning Helpful?mentioning

confidence: 99%

Robust Machine Comprehension Models via Adversarial Training

Wang¹,

Bansal²

2018

Proceedings of the 2018 Conference of the North American Chapter Of the Association for Computational Linguistics: Hu

View full text Add to dashboard Cite

It is shown that many published models for the Stanford Question Answering Dataset (Rajpurkar et al., 2016) lack robustness, suffering an over 50% decrease in F1 score during adversarial evaluation based on the AddSent (Jia and Liang, 2017) algorithm. It has also been shown that retraining models on data generated by AddSent has limited effect on their robustness. We propose a novel alternative adversary-generation algorithm, AddSentDiverse, that significantly increases the variance within the adversarial training data by providing effective examples that punish the model for making certain superficial assumptions. Further, in order to improve robustness to AddSent's semantic perturbations (e.g., antonyms), we jointly improve the model's semantic-relationship learning capabilities in addition to our AddSentDiversebased adversarial training data augmentation. With these additions, we show that we can make a state-of-the-art model significantly more robust, achieving a 36.5% increase in F1 score under many different types of adversarial evaluation while maintaining performance on the regular SQuAD task.

show abstract

“…Another common technique is DA (Zhang and Bansal, 2019) by artificially generating more questions to enhance the training data or in a MTL setup (Yatskar, 2018;Dhingra et al, 2018;. (Alberti et al, 2019a;Alberti et al, 2019b) combine models of question generation with answer extraction and filter results to ensure round-trip consistency to get the SOTA on NQ.…”

Section: Related Workmentioning

confidence: 99%

Towards building a Robust Industry-scale Question Answering System

Chakravarti

Ferritto

Iyer³

et al. 2020

Proceedings of the 28th International Conference on Computational Linguistics: Industry Track

View full text Add to dashboard Cite

Industry-scale NLP systems necessitate two features. 1. Robustness: "zero-shot transfer learning" (ZSTL) performance has to be commendable and 2. Efficiency: systems have to train efficiently and respond instantaneously. In this paper, we introduce the development of a production model called GAAMA (Go Ahead Ask Me Anything) which possess the above two characteristics. For robustness, it trains on the recently introduced Natural Questions (NQ) dataset. NQ poses additional challenges over older datasets like SQuAD: (a) QA systems need to read and comprehend an entire Wikipedia article rather than a small passage, and (b) NQ does not suffer from observation bias during construction, resulting in less lexical overlap between the question and the article. GAAMA consists of Attention-over-Attention, diversity among attention heads, hierarchical transfer learning, and synthetic data augmentation while being computationally inexpensive. Building on top of the powerful BERT QA model, GAAMA provides a ∼2.0% absolute boost in F 1 over the industry-scale state-of-the-art (SOTA) system on NQ. Further, we show that GAAMA transfers zero-shot to unseen real life and important domains as it yields respectable performance on two benchmarks: the BioASQ and the newly introduced CovidQA datasets.

show abstract

Simple and Effective Semi-Supervised Question Answering

Cited by 66 publications

References 23 publications

Unsupervised Question Answering by Cloze Translation

Unsupervised Question Answering by Cloze Translation

Robust Machine Comprehension Models via Adversarial Training

Towards building a Robust Industry-scale Question Answering System

Contact Info

Product

Resources

About