Metacognitive Adaptation to Enhance Lifelong Language Learning

Abstract

Lifelong language learning (LLL) aims at learning new tasks and retaining old tasks in the field of NLP. LAMOL is a recent LLL framework following data-free constraints. Previous works have been researched based on LAMOL with additional computing with more time costs or new parameters. However, they still have a gap between multi-task learning (MTL), which is regarded as the upper bound of LLL. In this paper, we propose Metacognitive Adaptation (Metac-Adapt) almost without adding additional time cost and computational resources to make the model generate better pseudo samples and then replay them. Experimental results demonstrate that Metac-Adapt is on par with MTL or better.

Lifelong Sequence Generation with Dynamic Module Expansion and Adaptation

“…• Metac-Adapt (Metac) (Wang et al, 2023) adapts LAMOL towards better semantic space for generating pseudo samples.…”