LAMOL: LAnguage MOdeling for Lifelong Language Learning

Sun, Fan-Keng; Ho, Cheng-Hao; Lee, Hung-Yi

doi:10.48550/arxiv.1909.03329

Cited by 14 publications

(18 citation statements)

References 15 publications

(18 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Yan et al (2021) proposes a new two-stage learning method that uses dynamic expandable representation for more effective incremental conceptual modelling. Among these methods, memory-based methods are the most effective in NLP tasks (Wang et al, 2019;Sun et al, 2019;d'Autume et al, 2019). Inspired by the success of memory-based methods in the field of NLP, we use the framework of memory replay to learn new relations that are constantly emerging.…”

Section: Continual Learningmentioning

confidence: 99%

Consistent Representation Learning for Continual Relation Extraction

Zhao¹,

Xu²,

Yang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Continual relation extraction (CRE) aims to continuously train a model on data with new relations while avoiding forgetting old ones. Some previous work has proved that storing a few typical samples of old relations and replaying them when learning new relations can effectively avoid forgetting. However, these memory-based methods tend to overfit the memory samples and perform poorly on imbalanced datasets. To solve these challenges, a consistent representation learning method is proposed, which maintains the stability of the relation embedding by adopting contrastive learning and knowledge distillation when replaying memory. Specifically, supervised contrastive learning based on a memory bank is first used to train each new task so that the model can effectively learn the relation representation. Then, contrastive replay is conducted of the samples in memory and makes the model retain the knowledge of historical relations through memory knowledge distillation to prevent the catastrophic forgetting of the old task. The proposed method can better learn consistent representations to alleviate forgetting effectively. Extensive experiments on FewRel and TACRED datasets show that our method significantly outperforms state-of-theart baselines and yield strong robustness on the imbalanced dataset. The code is publicly available at https://github.com/thuiar/ CRL.

show abstract

Section: Continual Learningmentioning

confidence: 99%

Consistent Representation Learning for Continual Relation Extraction

Zhao¹,

Xu²,

Yang³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Yan et al (2021) proposes a new two-stage learning method that uses dynamic expandable representation for more effective incremental conceptual modelling. Among these methods, memory-based methods are the most effective in NLP tasks (Wang et al, 2019;Sun et al, 2019;de Masson D'Autume et al, 2019). Inspired by the success of memory-based methods in the field of NLP, we use the framework of memory replay to learn new relations that are constantly emerging.…”

Section: Continual Learningmentioning

confidence: 99%

Consistent Representation Learning for Continual Relation Extraction

Zhao¹,

Xu²,

Yang³

et al. 2022

Findings of the Association for Computational Linguistics: ACL 2022

View full text Add to dashboard Cite

Continual relation extraction (CRE) aims to continuously train a model on data with new relations while avoiding forgetting old ones. Some previous work has proved that storing a few typical samples of old relations and replaying them when learning new relations can effectively avoid forgetting. However, these memory-based methods tend to overfit the memory samples and perform poorly on imbalanced datasets. To solve these challenges, a consistent representation learning method is proposed, which maintains the stability of the relation embedding by adopting contrastive learning and knowledge distillation when replaying memory. Specifically, supervised contrastive learning based on a memory bank is first used to train each new task so that the model can effectively learn the relation representation. Then, contrastive replay is conducted of the samples in memory and makes the model retain the knowledge of historical relations through memory knowledge distillation to prevent the catastrophic forgetting of the old task. The proposed method can better learn consistent representations to alleviate forgetting effectively. Extensive experiments on FewRel and TACRED datasets show that our method significantly outperforms state-of-the-art baselines and yield strong robustness on the imbalanced dataset. The code is publicly available at https://github.com/thuiar/CRL.

show abstract

“…These methods for LL mostly focus on tasks of the same type (referred as domains in this work). Recently, Sun et al (2019) proposes LAMOL, a general framework designed for lifelong language learning (LLL), where the model needs to continually learn from different domains as well as different types of NLP tasks.…”

Section: Lifelong Learningmentioning

confidence: 99%

“…More recent methods attempt to learn from different types of tasks. These include LAMOL (Sun et al, 2019) and its improvements (Chuang et al, 2020;Sun et al, 2020;Kanwatchara et al, 2021). Despite the effectiveness of these methods in LLL, there are several limitations.…”

Section: Introductionmentioning

confidence: 99%

LFPT5: A Unified Framework for Lifelong Few-shot Language Learning Based on Prompt Tuning of T5

Qin¹,

Joty²

2021

Preprint

View full text Add to dashboard Cite

Existing approaches to lifelong language learning rely on plenty of labeled data for learning a new task, which is hard to obtain in most real scenarios. Considering that humans can continually learn new tasks from a handful of examples, we expect the models also to be able to generalize well on new few-shot tasks without forgetting the previous ones. In this work, we define this more challenging yet practical problem as Lifelong Few-shot Language Learning (LFLL) and propose a unified framework for it based on prompt tuning of T5. Our framework called LFPT5 takes full advantage of PT's strong few-shot learning ability, and simultaneously trains the model as a task solver and a data generator. Before learning a new domain of the same task type, LFPT5 generates pseudo (labeled) samples of previously learned domains, and later gets trained on those samples to alleviate forgetting of previous knowledge as it learns the new domain. In addition, a KL divergence loss is minimized to achieve label consistency between the previous and the current model. While adapting to a new task type, LFPT5 includes and tunes additional prompt embeddings for the new task. With extensive experiments, we demonstrate that LFPT5 can be applied to various different types of tasks and significantly outperform previous methods in different LFLL settings.

show abstract

LAMOL: LAnguage MOdeling for Lifelong Language Learning

Cited by 14 publications

References 15 publications

Consistent Representation Learning for Continual Relation Extraction

Consistent Representation Learning for Continual Relation Extraction

Consistent Representation Learning for Continual Relation Extraction

LFPT5: A Unified Framework for Lifelong Few-shot Language Learning Based on Prompt Tuning of T5

Contact Info

Product

Resources

About