Representation Compensation Networks for Continual Semantic Segmentation

Abstract

We explore the capability of plain Vision Transformers (ViTs) for semantic segmentation using the encoder-decoder framework and introduce SegViTv2. In our work, we implement the decoder with the global attention mechanism inherent in ViT backbones and propose the light-weight Attentionto-Mask (ATM) module that effectively converts the global attention map into semantic masks for highquality segmentation results. Our decoder can outperform the most commonly-used decoder UpperNet in various ViT backbones while consuming only about 5% of the computational cost. For the encoder, we address the concern of the relatively high computational cost in the ViT-based encoders and propose a Shrunk++ structure that incorporates edge-aware query-based down-sampling (EQD) and query-based up-sampling (QU) modules. The Shrunk++ structure reduces the computational cost of the encoder by up to 50% while maintaining competitive performance. Furthermore, due to the flexibility of our ViT-based architecture, SegVit can be easily extended to semantic segmentation under the setting of continual learning, achieving nearly zero forgetting. Experiments show that our proposed SegViT outperforms recent segmentation methods on three popular benchmarks including ADE20k, COCO-Stuff-10k and PASCAL-Context

Continual Segment: Towards a Single, Unified and Accessible Continual Segmentation Model of 143 Whole-body Organs in CT Scans

“…CSWKD [41] weights the distillation loss based on the old and new class similarity. Other than knowledge distillation, RCIL [63] designs a two-branch module to decouple the representation learning of old and new classes. In multiorgan segmentation, only one study [31] applies CSS, based…”

“…Federated learning is a related solution [43], but it may not always be viable or easily accessible considering the requirement for sophisticated and expensive software/hardware computing infrastructures. Alternatively, we achieve this clinically preferred goal via continual semantic segmentation (CSS), which is emerging very recently in the natural image domain [5,11,36,37,63] but has been only scarcely studied for medical imaging [31,39].…”

“…There are several recent CSS work in computer vision [5,11,36,37,63]. MiB loss is often applied to handle the background-label conflicting issue [5,11].…”

Decomposed Knowledge Distillation for Class-Incremental Semantic Segmentation

“…Other techniques including mBCE and DKD in Sec. 3.2 can be applied to the off-the-shelf models, which brings significant performance gains, compared to current CISS methods [5,9,22,29] (See Tables 1 and 3).…”

“…Class-incremental semantic segmentation (CISS) adopts a CIL paradigm for the task of semantic segmentation. CISS methods [5,9,21,29] typically exploit a softmax cross-entropy (CE) term along with knowledge distillation (KD) [14]. Although the CE term helps to learn novel classes, applying the softmax function to all classes, including both old and novel ones, lowers class probabilities of old ones.…”