Reminding the incremental language model via data-free self-distillation

Metacognitive Adaptation to Enhance Lifelong Language Learning

“…In this section, we analyze whether the improvement of Metac-Adapt comes from generating better pseudo samples. We apply the CPS [8] to evaluate the quality of pseudo samples. The CPS is a BLEU-based method that calculates sample-averaged BLEU scores between the pseudo samples and the training dataset of each learned task to obtain the distribution of knowledge of learned tasks, and then calculates the Jensen-Shannon divergence between the BLEU distribution and the uniform distribution.…”

“…The CPS is a BLEU-based method that calculates sample-averaged BLEU scores between the pseudo samples and the training dataset of each learned task to obtain the distribution of knowledge of learned tasks, and then calculates the Jensen-Shannon divergence between the BLEU distribution and the uniform distribution. As described in [8], the lower the value of CPS represents better pseudo samples, which are beneficial to prevent catastrophic forgetting. CPS-n represents the quality of pseudo-data generated after learning n tasks.…”

“…Many works have been researched based on LAMOL. L2KD [6], DnR [7], DFSD [8] improved LAMOL by distilling the parts or all layer of the model. Rational-LAMOL [9] applied rationale information of samples to critical freezing parts of the module.…”