Self-Supervised Model Adaptation for Multimodal Semantic Segmentation

Abstract

Learning to reliably perceive and understand the scene is an integral enabler for robots to operate in the realworld. This problem is inherently challenging due to the multitude of object types as well as appearance changes caused by varying illumination and weather conditions. Leveraging complementary modalities can enable learning of semantically richer representations that are resilient to such perturbations. Despite the tremendous progress in recent years, most multimodal convolutional neural network approaches directly concatenate feature maps from individual modality streams rendering the model incapable of focusing only on the relevant complementary information for fusion. To address this limitation, we propose a mutimodal semantic segmentation framework that dynamically adapts the fusion of modalityspecific features while being sensitive to the object category, spatial location and scene context in a self-supervised manner. Specifically, we propose an architecture consisting of two modality-specific encoder streams that fuse intermediate encoder representations into a single decoder using our proposed self-supervised model adaptation fusion mechanism which optimally combines complementary features. As intermediate representations are not aligned across modalities, we introduce an attention scheme for better correlation. In addition, we propose a computationally efficient unimodal segmentation architecture termed AdapNet++ that incorporates a new encoder with multiscale residual units and an efficient atrous spatial pyramid pooling that has a lar-Abhinav Valada ger effective receptive field with more than 10× fewer parameters, complemented with a strong decoder with a multiresolution supervision scheme that recovers high-resolution details. Comprehensive empirical evaluations on Cityscapes, Synthia, SUN RGB-D, ScanNet and Freiburg Forest benchmarks demonstrate that both our unimodal and multimodal architectures achieve state-of-the-art performance while simultaneously being efficient in terms of parameters and inference time as well as demonstrating substantial robustness in adverse perceptual conditions.

Sparse and Noisy LiDAR Completion with RGB Guidance and Uncertainty

“…In fact, different fusion techniques can be considered: early fusion, late fusion or multi-level fusion. Valada et al [19] adopt the latter technique by extracting and combining feature maps at different stages in the encoder from multiple input streams. In general most works, such as [7,2], show that late fusion can achieve better performance.…”

DSNet: An Efficient CNN for Road Scene Segmentation

“…Cityscapes data Additional data mIoU (%) Runtime (s) DRN_CRL_Coarse [37] Fine, Coarse ImagNet 82.8 -DPC [3] Fine, Coarse ImageNet, COCO [17] 82.7 -RelationNet_Coarse [36] Fine, Coarse ImageNet 82.4 -SSMA [32] Fine…”

Small Obstacle Avoidance Based on RGB-D Semantic Segmentation

“…ICNet [28] achieves great balance between efficiency and accuracy by using a hierarchical structure to save time on high-resolution feature maps. As regards RGB-D semantic segmentation, some studies have tried to utilizing the depth information to achieve better segmentation accuracy [9,13,18,25]. Hazirbas [9] presented a fusion-based CNN architecture which is consisted of two encoder branches for RGB and depth channel.…”