SS-IL: Separated Softmax for Incremental Learning

Ahn, Hongjoon; Kwak, Jihwan; Lim, Subin; Bang, Hyeonsu; Kim, Hyojun; Moon, Taesup

doi:10.48550/arxiv.2003.13947

Cited by 1 publication

(3 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We also show that the mitigation of the old representation drift does not hinder the ability to learn and discriminate the new classes from the old ones. This property emerges from only learning the incoming data in isolation; as we will see, also isolating the rehearsal step (as in Ahn et al (2020)) leads to poor knowledge acquisition on the current task. Furthermore we show our ER-ACE objective can be combined with existing methods, leading to additional gains.…”

Section: Introductionmentioning

confidence: 90%

“…Hou et al (2019) considers addressing this imbalance through applying cosine similarity based loss as opposed to the typical cross entropy loss along with a distillation loss and a margin based loss with negatives mining to preserve the feature of previous classes. Recently, Ahn et al (2020) propose to learn the incoming tasks and the previous tasks separately. They use a masked softmax loss for the incoming and rehearsal data, to counter the class imbalance.…”

Section: Related Workmentioning

confidence: 99%

“…In this paper, we focus on the online setting, with potentially overlapping tasks. As we will see, work by Ahn et al (2020) developed to counter class imbalance, can inhibit learning of the current task in the online setting (see Appendix B). Lastly, Zeno et al ( 2018) uses a logit masking related to our method but their context is based on the multi-head setting, and does not consider replay based methods, where learning across tasks occurs.…”

Section: Related Workmentioning

confidence: 99%

See 2 more Smart Citations

New Insights on Reducing Abrupt Representation Change in Online Continual Learning

Caccia¹,

Aljundi²,

Asadi³

et al. 2022

Preprint

View full text Add to dashboard Cite

In the online continual learning paradigm, agents must learn from a changing distribution while respecting memory and compute constraints. Experience Replay (ER), where a small subset of past data is stored and replayed alongside new data, has emerged as a simple and effective learning strategy. In this work, we focus on the change in representations of observed data that arises when previously unobserved classes appear in the incoming data stream, and new classes must be distinguished from previous ones. We shed new light on this question by showing that applying ER causes the newly added classes' representations to overlap significantly with the previous classes, leading to highly disruptive parameter updates. Based on this empirical analysis, we propose a new method which mitigates this issue by shielding the learned representations from drastic adaptation to accommodate new classes. We show that using an asymmetric update rule pushes new classes to adapt to the older ones (rather than the reverse), which is more effective especially at task boundaries, where much of the forgetting typically occurs. Empirical results show significant gains over strong baselines on standard continual learning benchmarks 1 .

show abstract

Section: Introductionmentioning

confidence: 90%

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation