Uncertainty-Aware Contrastive Distillation for Incremental Semantic Segmentation

Abstract

A fundamental and challenging problem in deep learning is catastrophic forgetting, i.e. the tendency of neural networks to fail to preserve the knowledge acquired from old tasks when learning new tasks. This problem has been widely investigated in the research community and several Incremental Learning (IL) approaches have been proposed in the past years. While earlier works in computer vision have mostly focused on image classification and object detection, more recently some IL approaches for semantic segmentation have been introduced. These previous works showed that, despite its simplicity, knowledge distillation can be effectively employed to alleviate catastrophic forgetting. In this paper, we follow this research direction and, inspired by recent literature on contrastive learning, we propose a novel distillation framework, Uncertainty-aware Contrastive Distillation (UCD). In a nutshell, UCD is operated by introducing a novel distillation loss that takes into account all the images in a mini-batch, enforcing similarity between features associated to all the pixels from the same classes, and pulling apart those corresponding to pixels from different classes. In order to mitigate catastrophic forgetting, we contrast features of the new model with features extracted by a frozen model learned at the previous incremental step. Our experimental results demonstrate the advantage of the proposed distillation technique, which can be used in synergy with previous IL approaches, and leads to state-of-art performance on three commonly adopted benchmarks for incremental semantic segmentation. The code is available at https://github.com/ygjwd12345/UCD.

Federated Incremental Semantic Segmentation

“…We apply the constantly updated step-aware weight ψ t ij to the cross-entropy loss based on one-hot pseudo labels ỹt ij in Eq. (12). And the different forgetting paces can be alleviated by re-weighting gradient back-propagation of old classes (see Fig.…”

“…It is natural to seek effective incremental semantic segmentation (ISS) methods [7], [8], which can continually learn the model with the training samples of novel classes only. There have been some ISS works [9], [10], [11], [7], [8], [12], [13], which mainly focus on two key challenges, catastrophic forgetting [14] and background shift [9] (see Fig. 1).…”

Ecosystem Health of the Beiyun River Basin (Beijing, China) as Evaluated by the Analytic Hierarchy Process Method

“…3, the first category, known as data-replay methods, involves storing a portion of past training data as exemplar memory such as [26], [27], [28], [29], [30], [31], [32], [33], [34], [35], [36]. The second category, termed datafree methods, includes methods like [37], [38], [39], [40], [41], [42], [43], [44], [45], [46], [47], [48], [49]. These methods utilize transfer learning techniques, such as knowledge distillation, to inherit the capabilities of the old model.…”

“…Here we would like to discuss the advantage and necessity of continual learning based on specified models during the period of emerging large models. Although recent [26], [34], [35], [50], [51] Generative-replay Generative-data Generative-feature without storing real data, customized data replay heavy reliance on generative quality, high space complexity [28], [29], [32], [36], [52] Self-supervised Contrastive-learning Pseudo-labeling Foundation-model Driven strong adaptability, exemplar-memory free high training cost, hard to convergence [27], [41], [48], [53], [54] Regularization-based [39], [40], [43], [44], [55] Dynamic-architecture Parameter Allocation Architecture Decomposition Modular Network high model flexibility, high adaptability to diverse data network parameters gradually increases, high space complexity [30], [46], [56], [57], [58] large-model forms [59], [60] achieve fair zero-shot learning ability, they often lack the ability to classify targets with semantic understanding like humans. Another significant concern is cost.…”

“…The typical paradigm of this manner is introducing proxy tasks with objective functions. For example, contrastive learning can be set in feature or logits alignment [27], [41]. With respect to the inner feature distribution, SDR [27], PIFS [53] preserve classspecific prototypes as auxiliary supervision for contrastive learning during CL steps.…”