Generalizing Question Answering System with Pre-trained Language Model Fine-tuning

Abstract

With a large number of datasets being released and new techniques being proposed, Question answering (QA) systems have witnessed great breakthroughs in reading comprehension (RC) tasks. However, most existing methods focus on improving in-domain performance, leaving open the research question of how these models and techniques can generalize to out-ofdomain and unseen RC tasks. To enhance the generalization ability, we propose a multi-task learning framework that learns the shared representation across different tasks. Our model is built on top of a large pre-trained language model, such as XLNet, and then fine-tuned on multiple RC datasets. Experimental results show the effectiveness of our methods, with an average Exact Match score of 56.59 and an average F1 score of 68.98, which significantly improves the BERT-Large baseline by 8.39 and 7.22, respectively.

MRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehension

“…Harbin Institute of Technology HLTC (Su et al, 2019) Hong Kong University of Science & Technology BERT-cased-whole-word Aristo @ AI2 CLER (Takahashi et al, 2019) Fuji Xerox Co., Ltd. Adv. Train (Lee et al, 2019) 42Maru and Samsung Research BERT-Multi-Finetune Beijing Language and Culture University PAL IN DOMAIN University of California Irvine HierAtt (Osama et al, 2019) Alexandria University (Longpre et al, 2019) 82.3 68.5 66.9 74.6 70.8 FT XLNet 82.9 68.0 66.7 74.4 70.5 HLTC (Su et al, 2019) 81.0 65.9 65.0 72.9 69.0 BERT-cased-whole-word 79.4 61.1 61.4 71.2 66.3 CLER (Takahashi et al, 2019) 80.2 62.7 62.5 69.7 66.1 Adv. Train (Lee et al, 2019) 76.8 57.1 57.9 66.5 62.…”

“…Within these restrictions, we encouraged participants to explore how to best utilize the provided data. Inspired by Talmor and Berant (2019), two submissions (Su et al, 2019;Longpre et al, 2019) analyzed similarities between datasets. Unsurprisingly, the performance improved significantly when fine-tuned on the training dataset most similar to the evaluation dataset of interest.…”

“…Unsurprisingly, the performance improved significantly when fine-tuned on the training dataset most similar to the evaluation dataset of interest. Su et al (2019) found each of the development (Split II) datasets resembles one or two training datasets (Split I)-and thus training with all datasets is crucial for generalization across the multiple domains. They experimented with datafeeding methodologies, and found that shuffling instances of all six training datasets is more effective than sequentially feeding all examples from each dataset, one dataset after another.…”

Improving QA Generalization by Concurrent Modeling of Multiple Biases

“…We combine all the bias weights and use them to adapt the distillation loss. Su et al (2019) achieve considerable improvements by simply fine-tuning XLNet instead of BERT, and Longpre et al (2019) achieve further improvements by augmenting the training data with additional unanswerable questions.…”

“…Existing approaches to improve generalization in QA either are only applicable when there exist multiple training domains (Talmor and Berant, 2019;Takahashi et al, 2019; or rely on models and ensembles with larger capacity (Longpre et al, 2019;Su et al, 2019;. In contrast, our novel debiasing approach can be applied to both single and multi-domain scenarios, and it improves the model generalization without requiring larger pre-trained language models.…”