Gated Feedback Refinement Network for Dense Image Labeling

Abstract

Effective integration of local and global contextual information is crucial for semantic segmentation and dense image labeling. We develop two encoder-decoder based deep learning architectures to address this problem. We first propose a network architecture called Label Refinement Network (LRN) that predicts segmentation labels in a coarse-to-fine fashion at several spatial resolutions. In this network we also define loss functions at several stages to provide supervision at different stages of training. However, there are limits to the quality of refinement possible if ambiguous information is passed forward. In order to address this issue we also propose Gated Feedback Refinement Network (G-FRNet) that addresses this limitation. Initially, G-FRNet makes a coarse-grained prediction which it progressively refines to recover details by effectively integrating local and global contextual information during the refinement stages. This is achieved by gate units proposed in this work, that control information passed forward in order to resolve ambiguity. Experiments were conducted on four challenging dense labeling datasets (CamVid, PASCAL VOC 2012, Horse-Cow Parsing, PASCAL-Person-Part, and SUN-RGBD). G-FRNet achieves state-of-the-art semantic segmentation results on the CamVid and Horse-Cow Parsing datasets, and produces results competitive with the best performing approaches that appear in the literature for the other three datasets.

Recurrent Iterative Gating Networks for Semantic Segmentation

“…PASCAL VOC 2012 is a popular semantic segmentation dataset consisting of 1,464 images for training, 1,449 images for validation and 1,456 images for testing, which includes 20 object categories and one background class. Following prior work [7,35,20,32,7], we use the augmented training set that includes 10,582 images [14]. First, we report experimental results on the PASCAL VOC 2012 validation set.…”

“…There are a few specific considerations that motivate this paper, which presents a simple lightweight gating mechanism [42,20,28] that is top down wherein larger convolutional windows and more discriminative features play a role in guiding feedforward activation among earlier fea-…”

ESNet: An Efficient Symmetric Network for Real-Time Semantic Segmentation

“…Through adding skip connections, U-Net [25] designs an elegant symmetric network architecture, which stacks convolutional features from the encoder to the decoder activations. More recently, more attention have been paid to RefineNets [9,12,26,27], which adopt ResNet [2] in encoder-decoder structure, and have been demonstrated very effective on several semantic segmentation benchmarks [20,28].…”

Learning Fully Dense Neural Networks for Image Semantic Segmentation

“…The ablation experiment results are shown in Table 2. In order to further boost the gradient backpropagation and information flow, we compute multiple losses for different aggregated feature map F i motivated by (Zhao et al 2017;Islam et al 2017;Fu et al 2017). Specifically, F i is fed to upsample module to obtain a feature map L i with channel C, where C is number of classes in prediction labels.…”

“…And the aggregated feature map is feature maps from all the previous blocks. Thus, each feature map in the encoder has much shorter path to loss compared with previous encoder-decoder structure (Lin et al 2017a;Islam et al 2017). The gradient backpropagation and information flowing is much more efficient.…”