Multitask GANs for Semantic Segmentation and Depth Completion With Cycle Consistency

Zhang, Chongzhen; Tang, Yang; Zhao, Chaoqiang; Sun, Qiyu; Ye, Zhencheng; Kurths, Jürgen

doi:10.1109/tnnls.2021.3072883

Cited by 28 publications

(16 citation statements)

References 43 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Lee et al [18] introduced the Plane-Residual representation to interpret depth information and factorized the depth regression problem into a combination of discrete depth plane classification and plane-by-plane residual regression. Zhang et al [40] uses GANs to solve both semantic segmentation and depth completion tasks in outdoor scenarios. Cheng et al [3] proposed the convolutional spatial propagation network (CSPN) and generated the long-range context through a recurrent operation to lessen the burden of directly regressing the absolute depth information.…”

Section: Related Workmentioning

confidence: 99%

RGB-Depth Fusion GAN for Indoor Depth Completion

Wang¹,

Wang²,

Che³

et al. 2022

Preprint

View full text Add to dashboard Cite

The raw depth image captured by the indoor depth sensor usually has an extensive range of missing depth values due to inherent limitations such as the inability to perceive transparent objects and limited distance range. The incomplete depth map burdens many downstream vision tasks, and a rising number of depth completion methods have been proposed to alleviate this issue. While most existing methods can generate accurate dense depth maps from sparse and uniformly sampled depth maps, they are not suitable for complementing the large contiguous regions of missing depth values, which is common and critical. In this paper, we design a novel two-branch end-to-end fusion network, which takes a pair of RGB and incomplete depth images as input to predict a dense and completed depth map. The first branch employs an encoder-decoder structure to regress the local dense depth values from the raw depth map, with the help of local guidance information extracted from the RGB image. In the other branch, we propose an RGB-depth fusion GAN to transfer the RGB image to the fine-grained textured depth map. We adopt adaptive fusion modules named W-AdaIN to propagate the features across the two branches, and we append a confidence fusion head to fuse the two outputs of the branches for the final depth map. Extensive experiments on NYU-Depth V2 and SUN RGB-D demonstrate that our proposed method clearly improves the depth completion performance, especially in a more realistic setting of indoor environments with the help of the pseudo depth map.

show abstract

Section: Related Workmentioning

confidence: 99%

RGB-Depth Fusion GAN for Indoor Depth Completion

Wang¹,

Wang²,

Che³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Some methods narrow the distribution shift between different datasets in latent feature space [31], [35], [36], which are categorized as feature-based domain adaptation methods. Though these feature-based methods can perform well in classification tasks, they tend to fail in more complicated tasks like depth estimation and semantic segmentation [37], [38] . The development of Generative Adversarial Networks (GAN) [39] promotes the emergency of input-based [32], [40], [41] and output-based domain adaptation methods [13], [14], [42], [43].…”

Section: B Adversarial Domain Adaptationmentioning

confidence: 99%

Learn to Adapt for Monocular Depth Estimation

Sun¹,

Yen²,

Yang³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Monocular depth estimation is one of the fundamental tasks in environmental perception and has achieved tremendous progress in virtue of deep learning. However, the performance of trained models tends to degrade or deteriorate when employed on other new datasets due to the gap between different datasets. Though some methods utilize domain adaptation technologies to jointly train different domains and narrow the gap between them, the trained models cannot generalize to new domains that are not involved in training. To boost the transferability of depth estimation models, we propose an adversarial depth estimation task and train the model in the pipeline of meta-learning. Our proposed adversarial task mitigates the issue of meta-overfitting, since the network is trained in an adversarial manner and aims to extract domain invariant representations.In addition, we propose a constraint to impose upon cross-task depth consistency to compel the depth estimation to be identical in different adversarial tasks, which improves the performance of our method and smoothens the training process. Experiments demonstrate that our method adapts well to new datasets after few training steps during the test procedure.

show abstract

“…These methods utilize convolutional neural networks and combine sparse Li-DAR data with different modalities, e.g. RGB images [13], [14], semantic maps [16], [21], [22], and surface normal's [12]. These modalities act as guidance and significantly help in the recovery of missing depth values in the sparse maps.…”

Section: Introductionmentioning

confidence: 99%

SemAttNet: Towards Attention-based Semantic Aware Guided Depth Completion

Nazir¹,

Liwicki²,

Stricker³

et al. 2022

Preprint

View full text Add to dashboard Cite

Depth completion involves recovering a dense depth map from a sparse map and an RGB image. Recent approaches focus on utilizing color images as guidance images to recover depth at invalid pixels. However, color images alone are not enough to provide the necessary semantic understanding of the scene. Consequently, the depth completion task suffers from sudden illumination changes in RGB images (e.g., shadows). In this paper, we propose a novel three-branch backbone comprising color-guided, semanticguided, and depth-guided branches. Specifically, the color-guided branch takes a sparse depth map and RGB image as an input and generates color depth which includes color cues (e.g., object boundaries) of the scene. The predicted dense depth map of color-guided branch along-with semantic image and sparse depth map is passed as input to semantic-guided branch for estimating semantic depth. The depth-guided branch takes sparse, color, and semantic depths to generate the dense depth map. The color depth, semantic depth, and guided depth are adaptively fused to produce the output of our proposed three-branch backbone. In addition, we also propose to apply semantic-aware multi-modal attention-based fusion block (SAMMAFB) to fuse features between all three branches. We further use CSPN++ with Atrous convolutions to refine the dense depth map produced by our three-branch backbone. Extensive experiments show that our model achieves state-of-the-art performance in the KITTI depth completion benchmark at the time of submission.

show abstract

Multitask GANs for Semantic Segmentation and Depth Completion With Cycle Consistency

Cited by 28 publications

References 43 publications

RGB-Depth Fusion GAN for Indoor Depth Completion

RGB-Depth Fusion GAN for Indoor Depth Completion

Learn to Adapt for Monocular Depth Estimation

SemAttNet: Towards Attention-based Semantic Aware Guided Depth Completion

Contact Info

Product

Resources

About