Multi-Task Learning for Segmentation of Building Footprints with Deep Neural Networks

Abstract

The increased availability of high resolution satellite imagery allows to sense very detailed structures on the surface of our planet. Access to such information opens up new directions in the analysis of remote sensing imagery. However, at the same time this raises a set of new challenges for existing pixel-based prediction methods, such as semantic segmentation approaches. While deep neural networks have achieved significant advances in the semantic segmentation of high resolution images in the past, most of the existing approaches tend to produce predictions with poor boundaries. In this paper, we address the problem of preserving semantic segmentation boundaries in high resolution satellite imagery by introducing a new cascaded multi-task loss. We evaluate our approach on Inria Aerial Image Labeling Dataset which contains large-scale and high resolution images. Our results show that we are able to outperform state-of-the-art methods by 8.3% without any additional post-processing step.

Multi3Net: Segmenting Flooded Buildings via Fusion of Multiresolution, Multisensor, and Multitemporal Satellite Imagery

“…We used state-of-the-art pyramid sampling pooling (Zhao et al, 2017) to aggregate spatial context and found that this architecture outperformed fully convolutional networks (Maggiori et al, 2017b) and Mask-RCNNs (Ohleyer, 2018) on building footprint segmentation from very highresolution images. We showed that building footprint predictions obtained by only using publicly-available mediumresolution radar and optical satellite images in Multi 3 Net almost performs on par with building footprint segmentation models that use very high-resolution satellite imagery (Bischke et al, 2017). Building on this result, we used Multi 3 Net to segment flooded buildings, fusing multiresolution, multisensor, and multitemporal satellite imagery, and showed that full-fusion outperformed alternative fusion approaches.…”

Distance transform regression for spatially-aware deep semantic segmentation

“…It is also worth noting that our strategy succeeds while CRF did not improve classification results on this dataset as reported in Marmanis et al (2017). Bischke et al (2017). We report the overall accuracy (OA) and the intersection over union (IoU).…”

“…Inspired by this recent idea, we introduce a distance transform regression loss in a multi-task learning framework, which acts as a natural regularizer for semantic segmentation. This idea was tested independently from us in Bischke et al (2017), although only for building footprint extraction using a quantized distance transform that was roughly equivalent to standard multi-class classification task. Our method is simpler as it directly works on the distance transform using a true regression.…”

“…Note that several missing buildings are actually false positive in the ground truth. We also present a comparison to another multi-task approach which uses a distance transform Bischke et al (2017) in table 3, this time on their custom validation set. It shows that regression on SDT is better than SDT discretization followed by classification.…”

A Comparison and Strategy of Semantic Segmentation on Remote Sensing Images

“…And the decoder network first use max-pooling indices generated from the corresponding encoder to enhance the location information. In [18], Bischke et al use SegNet with a new cascaded multi-task loss to further preserve semantic segmentation boundaries in high resolution satellite images.…”