Tackling Catastrophic Forgetting and Background Shift in Continual Semantic Segmentation

Abstract

Deep learning approaches are nowadays ubiquitously used to tackle computer vision tasks such as semantic segmentation, requiring large datasets and substantial computational power. Continual learning for semantic segmentation (CSS) is an emerging trend that consists in updating an old model by sequentially adding new classes. However, continual learning methods are usually prone to catastrophic forgetting. This issue is further aggravated in CSS where, at each step, old classes from previous iterations are collapsed into the background. In this paper, we propose Local POD, a multi-scale pooling distillation scheme that preserves long-and short-range spatial relationships at feature level. Furthermore, we design an entropy-based pseudo-labelling of the background w.r.t. classes predicted by the old model to deal with background shift and avoid catastrophic forgetting of the old classes. Finally, we introduce a novel rehearsal method that is particularly suited for segmentation. Our approach, called PLOP, significantly outperforms state-of-the-art methods in existing CSS scenarios, as well as in newly proposed challenging benchmarks.

Pyramid Self-attention for Semantic Segmentation

“…The similarity between the response of the new model and that of the old model can be measured by cross-entropy loss if the response is the model output as in the LwF method [2] and the iCarL method [1], or more generally by Euclidean distance or cosine distance when the response is the output of single or multiple intermediate layers as in the LwM method [4] and the PODNet method [17]. Such knowledge distillation has been effectively applied to continual semantic segmentation [5], [6], [7], [10]. Besides knowledge distillation, replay strategy has also been proven very helpful for both continual classification [1], [18] and semantic segmentation [12].…”

“…In addition to alleviating the catastrophic forgetting issue, continual semantic segmentation also has to solve the background shifting issue [6], [7], [11], [12], i.e., background regions in images at one learning stage may contain regions of classes learned at another stage. Pseudo-labeling part of background regions as learned old classes from the new training images by the old model at each new learning stage has been shown helpful to alleviate this issue [7], [10], [12]. Aggregating the prediction probabilities of both the background class and new classes from the new class for background class during knowledge distillation can reduce the bias of background prediction between the old and the new model [6].…”

“…To alleviate the catastrophic forgetting issue, several strategies originally for continual classification tasks have been directly adopted and empirically proved effective as well for continual semantic segmentation. One strategy is knowledge distillation which transfers old knowledge by distilling the output of the last layer or multiple layers from the previously updated old model to the current new model [7], [10]. Another strategy is the usage of stored exemplars for each previously learned class when training the new model, allowing the new model to directly refresh old knowledge based on these limited old data.…”