A generalized multi-task learning approach to stereo DSM filtering in urban areas

Abstract

City models and height maps of urban areas serve as a valuable data source for numerous applications, such as disaster management or city planning. While this information is not globally available, it can be substituted by digital surface models (DSMs), automatically produced from inexpensive satellite imagery. However, stereo DSMs often suffer from noise and blur. Furthermore, they are heavily distorted by vegetation, which is of lesser relevance for most applications. Such basic models can be filtered by convolutional neural networks (CNNs), trained on labels derived from digital elevation models (DEMs) and 3D city models, in order to obtain a refined DSM. We propose a modular multi-task learning concept that consolidates existing approaches into a generalized framework. Our encoder-decoder models with shared encoders and multiple task-specific decoders leverage roof type classification as a secondary task and multiple objectives including a conditional adversarial term. The contributing single-objective losses are automatically weighted in the final multi-task loss function based on learned uncertainty estimates. We evaluated the performance of specific instances of this family of network architectures. Our method consistently outperforms the state of the art on common data, both quantitatively and qualitatively, and generalizes well to a new dataset of an independent study area.

Long-Short Skip Connections in Deep Neural Networks for DSM Refinement

“…Our previous approaches (Bittner et al, 2018(Bittner et al, , 2019b pursuit not only automatic height images creation from photogrammetric half-meter resolution satellite DSMs but also a simultaneous building shapes refinement on them involving cGANs. (Bittner et al, 2019a;Liebel et al, 2020) investigate mutli-task learning for improving the mutual information between different but correlated problems like roof type classification and building shape refinement. The work of (Bittner et al, 2019b) propose to incorporate several types of information, precisely intensity, and height, to gain both detailed roof ridge lines reconstruction and the compilation of building structure if they are badly represented in photogrammetric DSMs.…”

“…Several experiments have been already done by us for obtaining multiple remote sensing tasks from deep network architecture applying this methodology (Liebel et al, 2020). Based on these experiments, we have decided to incorporate the learning of balancing hyper-parameters wl in Eq.…”

“…The starting values of balancing hyper-parameters in Eq. (6) were equally initialized with 0.3, making wcLSGAN unchangeable and wL 1 , wnormal learnable, as proposed by (Liebel et al, 2020). The network was trained on a single NVIDIA TITAN X (PASCAL) GPU with 12 GB of memory.…”

“…Munich dataset: Even though Munich and Berlin belong to the same country and their building appearances can have a similar style, it is impossible to meet identical building constructions. Since there is a time difference of several years between our satellite images and CityGML data, many inconsistencies in terms of newly built or demolished city constructions exist over the available scene (Liebel et al, 2020). Therefore, we select a smaller area depicted in Fig.…”

Multitask Learning for Human Settlement Extent Regression and Local Climate Zone Classification

“…As already suggested by the definition of the two schemes, prior work also indicated that HSE and LCZ have close correspondence for different study areas [1], [6]. To exploit the complementary nature of the HSE regression and LCZ classification tasks, we propose a multitask learning (MTL) framework to jointly predict HSE and LCZ, considering that MTL has been shown to be a powerful technique for improving model generalization by leveraging domain knowledge of related complementary tasks [7], [8]. In this work, we present a feature-based MTL system that mainly consists of a shared backbone network to capture a common representation for both HSE regression and LCZ classification, and soft-attention modules to adaptively select task-specific features.…”