Efficient RGB-D Semantic Segmentation for Indoor Scene Analysis

Seichter, Daniel; Köhler, Mona; Lewandowski, Benjamin; Wengefeld, Tim; Groß, Horst–Michael

doi:10.1109/icra48506.2021.9561675

Cited by 149 publications

(62 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…First, our method heavily relies on semantic segmentation method, bad segmentation will lead to awful geometry and interlaced texture. We plan to address it by incorporating the efficient RGBD segmentation [40]. Besides, since depth sensors lack observations from specific materials such as shaggy hair and yarn clothes, our method cannot get good geometry to these areas.…”

Section: Limitationmentioning

confidence: 99%

NeuralHOFusion: Neural Volumetric Rendering under Human-object Interactions

Jiang¹,

Jiang²,

Sun³

et al. 2022

Preprint

View full text Add to dashboard Cite

4D modeling of human-object interactions is critical for numerous applications. However, efficient volumetric capture and rendering of complex interaction scenarios, especially from sparse inputs, remain challenging. In this paper, we propose NeuralFusion, a neural approach for volumetric human-object capture and rendering using sparse consumer RGBD sensors. It marries traditional non-rigid fusion with recent neural implicit modeling and blending advances, where the captured humans and objects are layerwise disentangled. For geometry modeling, we propose a neural implicit inference scheme with non-rigid key-volume fusion, as well as a template-aid robust object tracking pipeline. Our scheme enables detailed and complete geometry generation under complex interactions and occlusions. Moreover, we introduce a layer-wise human-object texture rendering scheme, which combines volumetric and image-based rendering in both spatial and temporal domains to obtain photo-realistic results. Extensive experiments demonstrate the effectiveness and efficiency of our approach in synthesizing photo-realistic free-view results under complex human-object interactions.

show abstract

Section: Limitationmentioning

confidence: 99%

NeuralHOFusion: Neural Volumetric Rendering under Human-object Interactions

Jiang¹,

Jiang²,

Sun³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Unlike other approaches [2, 20 -23], we do not apply any post-processing to the resulting maps, such as CRFs, as this would notably increase runtime. For semantic segmentation, we rely on our recently proposed RGB-D approach for efficient indoor scene analysis, called ESANet [8]. Semantic mapping can be performed using either semantic Bayesian kernel inference mapping (S-BKI), the proposed optimized version of this approach (OS-BKI), or the proposed semantic…”

Section: Efficient and Robust Semantic Mappingmentioning

confidence: 99%

“…Incorporating depth images can alleviate this effect by providing complementary geometric information. In [8], we have shown that RGB-D segmentation is superior to processing RGB images solely and that it can still be performed in real time if the network architecture is carefully designed. More precisely, we use ESANet-R34-NBt1D (enhanced ResNet34-based encoder utilizing the Non-Bottleneck-1D block (NBt1D) [27]) in our pipeline.…”

Section: A Semantic Segmentationmentioning

confidence: 99%

“…Moreover, it is to allow our robots to be aware of the current scene, i.e., by not blocking doors or the line of sight to objects such as a TV. To match the character of these dynamic environments, we build upon a strong long-term localization determined via RTAB-Map [7] and build shortterm semantic 3D representations of the current environment Semantic occupancy NDT (S-NDT) map built with predicted segmentation of ESANet [8] and cell size of 10cm for scene ai 048 010 of the Hypersim [9] test set. Right: for the given pose counterclockwise from bottom: color image, ground-truth annotation (GT), semantic segmentation output of ESANet (Seg.…”

Section: Introductionmentioning

confidence: 99%

“…We first evaluate their capability to represent semantics by using the provided ground-truth annotation for mapping, i.e., perfect data is assumed. Subsequently, we combine the mapping with our efficient RGB-D semantic segmentation approach [8] in order to allow a comparison in a more realistic setting when semantic information is noisy. Finally, we show qualitative results of our complete pipeline for efficient and robust semantic mapping in one of our domestic environments and, thus, give an impression of how it performs on real data.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Efficient and Robust Semantic Mapping for Indoor Environments

Seichter¹,

Langer²,

Wengefeld³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

A key proficiency an autonomous mobile robot must have to perform high-level tasks is a strong understanding of its environment. This involves information about what types of objects are present, where they are, what their spatial extend is, and how they can be reached, i.e., information about free space is also crucial. Semantic maps are a powerful instrument providing such information. However, applying semantic segmentation and building 3D maps with high spatial resolution is challenging given limited resources on mobile robots. In this paper, we incorporate semantic information into efficient occupancy normal distribution transform (NDT) maps to enable real-time semantic mapping on mobile robots. On the publicly available dataset Hypersim, we show that, due to their sub-voxel accuracy, semantic NDT maps are superior to other approaches. We compare them to the recent state-of-the-art approach based on voxels and semantic Bayesian spatial kernel inference (S-BKI) and to an optimized version of it derived in this paper. The proposed semantic NDT maps can represent semantics to the same level of detail, while mapping is 2.7 to 17.5 times faster. For the same grid resolution, they perform significantly better, while mapping is up to more than 5 times faster. Finally, we prove the real-world applicability of semantic NDT maps with qualitative results in a domestic application.

show abstract

Depth-Adapted CNN for RGB-D Cameras

Allibert

Stolz

et al. 2021

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Recent RGB-D semantic segmentation has motivated research interest thanks to the accessibility of complementary modalities from the input side. Existing works often adopt a two-stream architecture that processes photometric and geometric information in parallel, with few methods explicitly leveraging the contribution of depth cues to adjust the sampling position on RGB images. In this paper, we propose a novel framework to incorporate the depth information in the RGB convolutional neural network (CNN), termed Z-ACN (Depth-Adapted CNN). Specifically, our Z-ACN generates a 2D depthadapted offset which is fully constrained by low-level features to guide the feature extraction on RGB images. With the generated offset, we introduce two intuitive and effective operations to replace basic CNN operators: depth-adapted convolution and depth-adapted average pooling. Extensive experiments on both indoor and outdoor semantic segmentation tasks demonstrate the effectiveness of our approach.

show abstract

Efficient RGB-D Semantic Segmentation for Indoor Scene Analysis

Cited by 149 publications

References 29 publications

NeuralHOFusion: Neural Volumetric Rendering under Human-object Interactions

NeuralHOFusion: Neural Volumetric Rendering under Human-object Interactions

Efficient and Robust Semantic Mapping for Indoor Environments

Depth-Adapted CNN for RGB-D Cameras

Contact Info

Product

Resources

About