AlignSeg: Feature-Aligned Segmentation Networks

Abstract

Aggregating features in terms of different convolutional blocks or contextual embeddings has been proven to be an effective way to strengthen feature representations for semantic segmentation. However, most of current popular network architectures tend to ignore the misalignment issues during the feature aggregation process caused by 1) step-by-step downsampling operations, and 2) indiscriminate contextual information fusion. In this paper, we explore the principles in addressing such feature misalignment issues and inventively propose Feature-Aligned Segmentation Networks (AlignSeg). AlignSeg consists of two primary modules, i.e., the Aligned Feature Aggregation (AlignFA) module and the Aligned Context Modeling (AlignCM) module. First, AlignFA adopts a simple learnable interpolation strategy to learn transformation offsets of pixels, which can effectively relieve the feature misalignment issue caused by multiresolution feature aggregation. Second, with the contextual embeddings in hand, AlignCM enables each pixel to choose private custom contextual information in an adaptive manner, making the contextual embeddings aligned better to provide appropriate guidance. We validate the effectiveness of our AlignSeg network with extensive experiments on Cityscapes and ADE20K, achieving new state-of-the-art mIoU scores of 82.6% and 45.95%, respectively. Our source code will be made available.

Stage-Aware Feature Alignment Network for Real-Time Semantic Segmentation of Street Scenes

“…Unfortunately, the segmentation accuracy of these methods on street scene images, which usually contain many small objects (such as poles and traffic lights), is still far from being satisfactory. This is partly due to the fact that these methods ignore the misalignment between different levels of feature maps, which may lead to the misclassification of boundaries for small objects [23].…”

“…For example, Guided Upsampling Network [24] adopts a guided upsampling module to enrich upsampling operators by learning a transformation based on high-resolution inputs. Huang et al [23] propose the Feature-Aligned Segmentation Networks (AlignNet), which mainly consist of an Aligned Feature Aggregation module (AlignFA) and an Aligned Context Modeling module (AlignCM), to deal with the misalignment problem. Similarly, Semantic Flow Network (SFNet) [25] develops the Flow Alignment Module (FAM) to align and aggregate different levels of feature maps.…”

Half-Real Half-Fake Distillation for Class-Incremental Semantic Segmentation

“…Current state-of-the-art semantic segmentation approaches based on the fully convolutional network (FCN) [23] have made remarkable progress in several ways, e.g. by modeling context information [50,7,49,43,19], recovering the spatial details [8,36,20] or designing stronger networks [46,42,34]. The vast majority of semantic segmentation methods consider a static setting, i.e., the training data for all classes are available before training.…”

“…Semantic segmentation, aims at assigning semantic class labels to each pixel in a given image, provides significant impacts on various real-world applications, such as autonomous driving [14], augmented reality [1], etc. Specifically, current state-of-the-art semantic segmentation approaches based on the fully convolutional network (FCN) [23] have made remarkable progress in several ways, e.g., by modeling context information [50,7,49,43,19], recovering the spatial details [8,36,20] or designing stronger * equal contribution networks [46,42,34].…”

High-Resolution Deep Image Matting

“…Recently, deep learning has achieved great success in many computer vision tasks, such as in image classification [10,21], object detection [30,43], semantic segmentation [5,19] and deep learning applications in medicine [42,41,40] and agriculture [7,6] etc. Following this great success, the adoption of deep learning in image matting has also been widely explored in the past few years.…”