Semantic Segmentation via Pixel-to-Center Similarity Calculation

Abstract

Since the fully convolutional network has achieved great success in semantic segmentation, lots of works have been proposed focusing on extracting discriminative pixel feature representations. However, we observe that existing methods still suffer from two typical challenges, i.e. (i) large intra-class feature variation in different scenes, (ii) small inter-class feature distinction in the same scene. In this paper, we first rethink semantic segmentation from a perspective of similarity between pixels and class centers. Each weight vector of the segmentation head represents its corresponding semantic class in the whole dataset, which can be regarded as the embedding of the class center. Thus, the pixel-wise classification amounts to computing similarity in the final feature space between pixels and the class centers. Under this novel view, we propose a Class Center Similarity layer (CCS layer) to address the above-mentioned challenges by generating adaptive class centers conditioned on different scenes and supervising the similarities between class centers. It utilizes a Adaptive Class Center Module (ACCM) to generate class centers conditioned on each scene, which adapt the large intra-class variation between different scenes. Specially designed loss functions are introduced to control both inter-class and intra-class distances based on predicted center-to-center and pixel-to-center similarity, respectively. Finally, the CCS layer outputs the processed pixel-to-center similarity as the segmentation prediction. Extensive experiments demonstrate that our model performs favourably against the state-of-the-art CNN-based methods.

Beyond Prototypes: Semantic Anchor Regularization for Better Representation Learning

“…In addition, we only use and update semantic anchors when it is correctly classified with a probability greater than δ for better inter-class separation. In summary, the above training strategy ensures the independence of learning between semantic anchors and pixel features, even though the main task and auxiliary task share the same classifier, which is inherently different from previous works (Huang et al 2022;Wu et al 2023;Hu, Cui, and Wang 2021) Overall. Integrating all components, the overall loss for SAR representation learning is the weighted sum of the presented loss components,…”