Two Stream 3D Semantic Scene Completion

Garbade, Martin; Chen, Yueh-Tung; Sawatzky, Johann; Gall, Jüergen

doi:10.48550/arxiv.1804.03550

Cited by 6 publications

(17 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In order to incorporating the color information, TS3D [13] introduces the RGB image into SSC and uses a 2D network to acquire semantic segmentation results. Semantic outputs of the RGB stream are concatenated with inputs of the depth stream to obtain the completed 3D scene.…”

Section: Semantic Scene Completionmentioning

confidence: 99%

“…Since SSCNet only employs depth as input, the proposed GRFNet which use RGB and depth information achieves much more accurate results. Although TS3D [13] and DDR-SSC use both RGB and depth information, these methods only adopt simple fusion strategy. On the contrary, GRFNet benefited from the novel fusion block, to obtain 0.7% and 2.5% improvements compared to DDR-SSC, and 5.9% and 3.2% improvements compared to TS3D for SC and SSC tasks respectively.…”

Section: Quantitative Analysismentioning

confidence: 99%

“…Many vision applications have already benefit from using both modalities in their tasks, such as object detection [16,3], video segmentation [12,36,10], action recognition [21,20,42], or visual SLAM [23,40,29]. Recent studies [13,25] in SSC also demonstrate that employing both, RGB image and depth, can outperform using only one modality [35].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

3D Gated Recurrent Fusion for Semantic Scene Completion

Liu¹,

Li²,

Yan³

et al. 2020

Preprint

View full text Add to dashboard Cite

This paper tackles the problem of data fusion in the semantic scene completion (SSC) task, which can simultaneously deal with semantic labeling and scene completion. RGB images contain texture details of the object(s) which are vital for semantic scene understanding. Meanwhile, depth images capture geometric clues of high relevance for shape completion. Using both RGB and depth images can further boost the accuracy of SSC over employing one modality in isolation. We propose a 3D gated recurrent fusion network (GRFNet), which learns to adaptively select and fuse the relevant information from depth and RGB by making use of the gate and memory modules. Based on the single-stage fusion, we further propose a multi-stage fusion strategy, which could model the correlations among different stages within the network. Extensive experiments on two benchmark datasets demonstrate the superior performance and the effectiveness of the proposed GRFNet for data fusion in SSC. Code will be made available.

show abstract

Section: Semantic Scene Completionmentioning

confidence: 99%

Section: Quantitative Analysismentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

3D Gated Recurrent Fusion for Semantic Scene Completion

Liu¹,

Li²,

Yan³

et al. 2020

Preprint

View full text Add to dashboard Cite

show abstract

“…Garbade et al [6] proposed a two-stream approach that jointly leverages the depth and visual information. In specific, it first constructs an incomplete 3D semantic tensor for the inferred 2D semantic information, and then adopts a vanilla 3D CNN to infer the complete 3D semantic tensor.…”

Section: Semantic Scene Completionmentioning

confidence: 99%

“…Recently, promising progress has been achieved for SSC [16,6,8,10,13] by employing deep convolutional neural networks (CNNs). A direct solution is to use 3D convolutional neural network [16] to model the volumetric context, which consists of a stack of conventional 3D convolutional layers.…”

Section: Introductionmentioning

confidence: 99%

Anisotropic Convolutional Networks for 3D Semantic Scene Completion

Han

Wang

et al. 2020

Preprint

View full text Add to dashboard Cite

As a voxel-wise labeling task, semantic scene completion (SSC) tries to simultaneously infer the occupancy and semantic labels for a scene from a single depth and/or RGB image. The key challenge for SSC is how to effectively take advantage of the 3D context to model various objects or stuffs with severe variations in shapes, layouts and visibility. To handle such variations, we propose a novel module called anisotropic convolution, which properties with flexibility and power impossible for the competing methods such as standard 3D convolution and some of its variations. In contrast to the standard 3D convolution that is limited to a fixed 3D receptive field, our module is capable of modeling the dimensional anisotropy voxel-wisely. The basic idea is to enable anisotropic 3D receptive field by decomposing a 3D convolution into three consecutive 1D convolutions, and the kernel size for each such 1D convolution is adaptively determined on the fly. By stacking multiple such anisotropic convolution modules, the voxel-wise modeling capability can be further enhanced while maintaining a controllable amount of model parameters. Extensive experiments on two SSC benchmarks, NYU-Depth-v2 and NYUCAD, show the superior performance of the proposed method. Our code is available at https://waterljwant.github.io/SSC/ .

show abstract

Deep Reinforcement Learning of Volume-Guided Progressive View Inpainting for 3D Point Scene Completion From a Single Depth Image

Han

Cui

Zhang

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

View full text Add to dashboard Cite

We present a deep reinforcement learning method of progressive view inpainting for 3D point scene completion under volume guidance, achieving high-quality scene reconstruction from only a single depth image with severe occlusion. Our approach is end-to-end, consisting of three modules: 3D scene volume reconstruction, 2D depth map inpainting, and multi-view selection for completion. Given a single depth image, our method first goes through the 3D volume branch to obtain a volumetric scene reconstruction as a guide to the next view inpainting step, which attempts to make up the missing information; the third step involves projecting the volume under the same view of the input, concatenating them to complete the current view depth, and integrating all depth into the point cloud. Since the occluded areas are unavailable, we resort to a deep Q-Network to glance around and pick the next best view for large hole completion progressively until a scene is adequately reconstructed while guaranteeing validity. All steps are learned jointly to achieve robust and consistent results. We perform qualitative and quantitative evaluations with extensive experiments on the SUNCG data, obtaining better results than the state of the art.

show abstract

Two Stream 3D Semantic Scene Completion

Cited by 6 publications

References 0 publications

3D Gated Recurrent Fusion for Semantic Scene Completion

3D Gated Recurrent Fusion for Semantic Scene Completion

Anisotropic Convolutional Networks for 3D Semantic Scene Completion

Deep Reinforcement Learning of Volume-Guided Progressive View Inpainting for 3D Point Scene Completion From a Single Depth Image

Contact Info

Product

Resources

About