Continual Domain Adaptation for Machine Reading Comprehension

Abstract

Machine reading comprehension (MRC) has become a core component in a variety of natural language processing (NLP) applications such as question answering and dialogue systems. It becomes a practical challenge that an MRC model needs to learn in non-stationary environments, in which the underlying data distribution changes over time. A typical scenario is the domain drift, i.e. different domains of data come one after another, where the MRC model is required to adapt to the new domain while maintaining previously learned ability. To tackle such a challenge, in this work, we introduce the Continual Domain Adaptation (CDA) task for MRC. So far as we know, this is the first study on the continual learning perspective of MRC. We build two benchmark datasets for the CDA task, by reorganizing existing MRC collections into different domains with respect to context type and question type, respectively. We then analyze and observe the catastrophic forgetting (CF) phenomenon of MRC under the CDA setting. To tackle the CDA task, we propose several BERT-based continual learning MRC models using either regularization-based methodology or dynamic-architecture paradigm. We analyze the performance of different continual learning MRC models under the CDA task and show that the proposed dynamic-architecture based model achieves the best performance. CCS CONCEPTS • Information systems → Question answering.

Less Is More: Domain Adaptation with Lottery Ticket for Reading Comprehension

“…Talmor and Berant (2019), Khashabi et al (2020) and Lourie et al (2021) improve the generalization by training on multiple datasets. Su et al (2020) introduces Adapters (Houlsby et al, 2019) to accommodate each domain. Theses method requires a quite amount of annotated data to work.…”

Continual Machine Reading Comprehension via Uncertainty-aware Fixed Memory and Adversarial Domain Adaptation

“…Existing studies for continual MRC can mainly be divided into three categories. The first class is model expansion techniques that design domainindividual classifier for each in coming domain (Su et al, 2020). However, it is expensive and unpractical in real-world.…”

“…Few previous studies apply continual learning to MRC. Su et al (2020) adapted EWC method and enlarged the MRC architecture when a new domain arrives. Su et al (2020) added a penalty regularization that restricts the change of important parameters to prevent forgetting.…”

“…Su et al (2020) adapted EWC method and enlarged the MRC architecture when a new domain arrives. Su et al (2020) added a penalty regularization that restricts the change of important parameters to prevent forgetting. de Masson d'Autume et al (2019) and (Abujabal et al, 2018) designed episodic memory based methods that store training samples from previously seen data, which are later rehearsed to learn new domains.…”

“…In this scenario, the MRC system can only trained on the latest coming domain data without access the previous seen data. To tackle this issue, if we directly fine-tune the MRC model on each new incoming domain, the performance on earlier domains will significantly drop (Su et al, 2020). Another naive approach is to retrain the whole MRC model from scratch, but it is costly and time-consuming.…”