Classification with an edge: Improving semantic image segmentation with boundary detection

Abstract

We present an end-to-end trainable deep convolutional neural network (DCNN) for semantic segmentation with built-in awareness of semantically meaningful boundaries. Semantic segmentation is a fundamental remote sensing task, and most state-of-the-art methods rely on DCNNs as their workhorse. A major reason for their success is that deep networks learn to accumulate contextual information over very large receptive fields. However, this success comes at a cost, since the associated loss of effective spatial resolution washes out high-frequency details and leads to blurry object boundaries. Here, we propose to counter this effect by combining semantic segmentation with semantically informed edge detection, thus making class boundaries explicit in the model. First, we construct a comparatively simple, memory-efficient model by adding boundary detection to the segnet encoder-decoder architecture. Second, we also include boundary detection in fcn-type models and set up a high-end classifier ensemble. We show that boundary detection significantly improves semantic segmentation with CNNs in an end-to-end training scheme. Our best model achieves > 90% overall accuracy on the ISPRS Vaihingen benchmark.

High-Resolution Remote Sensing Data Classification over Urban Areas Using Random Forest Ensemble and Fully Connected Conditional Random Field

“…Compared to the best CNN-based methods, namely "CASIA2" and "DLR_10", our "NLPR" still has a notable gap. However, to get a higher performance, an extremely deep and complicated network with 101 layers is used by "CASIA2", several deep networks with different structures are combined in "DLR_10" [69], and the quality of NDSM used in "DLR_10" [69] is also higher than ours. We think this gap is reasonable.…”

Segment-before-Detect: Vehicle Detection and Classification through Semantic Segmentation of Aerial Images

“…Using a simple morphological approach for connected components extraction, we showed that these high resolution semantic maps were sufficient to extract object-level boundaries that outperform traditional vehicle detection methods. Future work on this step could involve integrating contour prediction in the segmentation network [34] or moving to direct instance prediction using recurrent attention [39,44].…”

“…However, predictions from SegNet can be noisy, as CNN tends to have blurred transitions between classes [34]. Therefore, to alleviate perturbations in the predictions coming out of the network, we first operate morphological opening with a small radius to erode the vehicle mask.…”

ERN: Edge Loss Reinforced Semantic Segmentation Network for Remote Sensing Images

“…Traditional unsupervised shadow detection methods [60] often failed in the dark regions, such as black cars and roofs. Therefore, we manually labeled the shadow masks for the test tiles (areas: 11,15,28,30,34) in ISPRS Vaihingen Dataset [35]. We first pre-processed the original images with the contrast preserving decolorization technology [61], which can help human better separate shadow regions from surroundings.…”

“…Cheng et al [33] fused semantic segmentation net and edge net with a regularization method to refine entire network. Marmanis et al [34] proposed a model that cascades the edge net (HED [29]) and semantic segmentation net (FCN [17]/SegNet [19]), where the model is complex and the training phase must be carefully fine-tuned. In contrast to these works, we sought to establish a simple and scalable model that integrates multiple weighted edge structures into semantic segmentation.…”