On the Effectiveness of Weakly Supervised Semantic Segmentation for Building Extraction From High-Resolution Remote Sensing Imagery

Abstract

A critical obstacle to achieve semantic segmentation of remote sensing images by the deep convolutional neural network is the requirement of huge pixel-level labels. Taking building extraction as an example, this study focuses on how to effectively apply weakly supervised semantic segmentation (WSSS) to highresolution remote sensing (HR) images with image-level labels, which is a prominent solution for the huge labeling challenge. The widely-used two-step WSSS framework is adopted, in which the pseudo-masks are first produced from image-level labels and followed by a segmentation network trained by the pseudo-masks. In addition, the fully connected conditional random field (CRF) is utilized to explore spatial context in both training and prediction stages. Detailed analyses are implemented on applying WSSS on HR images in terms of producing pseudo-masks, training segmentation network, and optimizing predictions. We show that the trade-off between precision and recall of pseudo-masks, as well as the boundary accuracy and the background, needs to be carefully considered. The benefits of the segmentation network in the two-step framework are demonstrated in comparison to using classification network only for WSSS, and the effects of CRF-loss are identified to be powerful for improving the segmentation network while not appropriate for dense buildings. An overlapping strategy and CRF post-processing are further demonstrated to be effective for optimizing the segmentation results during inferencing. Through deliberate settings, we can generate results comparable to fully supervised on the ISPRS Potsdam and Vaihingen dataset, which is meaningful for promoting WSSS applications for extracting geographic information from HR images.

Improved Pseudomasks Generation for Weakly Supervised Building Extraction From High-Resolution Remote Sensing Imagery

“…In SPMF-Net, a superpixel pooling layer was added to the classification network to improve the integrity and boundary accuracy of a detected building. A two-step training strategy approach was derived by [13], in which the fully connected conditional random field (CRF) was utilized to explore the spatial context in both training and prediction stages.…”

“…Although those methods have made significant improvements in building extraction, their performances are highly dependent on the quality of pseudo-masks [13]. Due to the CAMs focusing on the most discriminative parts, the generated pseudo-masks have a big gap with the ground-truth (GT) of buildings.…”

“…Following the data pre-processing in previous works [13], the raw images in the training dataset are cropped into 256 × 256 patches with a sliding step size of 128. And the image-level labels in this study are similarly determined by the pixel ratio of buildings in an image patch.…”

“…For training the segmentation network, the value of batch size was set to 8, and the number of classes was set to 2. In its inference process, the testing images were evaluated by an overlapping strategy [13].…”

“…The approach generated CAMs that retain the shape and boundary information in superpixel and contributed to accurate extraction of buildings. Li et al [13] adopted conditional random field (CRF) to optimize CAMs obtained by classification network and introduced CRF-loss [14] and CRF postprocessing in the segmentation network for accurate building extraction. For WSSS methods, generating high-quality pseudo-masks is crucial for accurate building extraction.…”