SCFusion: Real-time Incremental Scene Reconstruction with Semantic Completion

Wu, Shuncheng; Tateno, Keisuke; Navab, Nassir; Tombari, Federico

doi:10.1109/3dv50981.2020.00090

Cited by 22 publications

(4 citation statements)

References 43 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In image-based techniques for semantic scene completion, many proposed methods in literature employ convolutional neural networks or transformers to process multi-modal data, like RGB and depth images [1,[8][9][10][11][12][13][14][15][16][17][18][19][20]. For instance, Song et al [1] proposed an end-to-end 3D convolutional network, introducing a dilation-based context module for efficient large-receptive-field context learning.…”

Section: Image-based Methodsmentioning

confidence: 99%

Voxel- and Bird’s-Eye-View-Based Semantic Scene Completion for LiDAR Point Clouds

Liang,

Akhtar,

Vice

et al. 2024

Remote Sensing

View full text Add to dashboard Cite

Semantic scene completion is a crucial outdoor scene understanding task that has direct implications for technologies like autonomous driving and robotics. It compensates for unavoidable occlusions and partial measurements in LiDAR scans, which may otherwise cause catastrophic failures. Due to the inherent complexity of this task, existing methods generally rely on complex and computationally demanding scene completion models, which limits their practicality in downstream applications. Addressing this, we propose a novel integrated network that combines the strengths of 3D and 2D semantic scene completion techniques for efficient LiDAR point cloud scene completion. Our network leverages a newly devised lightweight multi-scale convolutional block (MSB) to efficiently aggregate multi-scale features, thereby improving the identification of small and distant objects. It further utilizes a layout-aware semantic block (LSB), developed to grasp the overall layout of the scene to precisely guide the reconstruction and recognition of features. Moreover, we also develop a feature fusion module (FFM) for effective interaction between the data derived from two disparate streams in our network, ensuring a robust and cohesive scene completion process. Extensive experiments with the popular SemanticKITTI dataset demonstrate that our method achieves highly competitive performance, with an mIoU of 35.7 and an IoU of 51.4. Notably, the proposed method achieves an mIoU improvement of 2.6 % compared to previous methods.

show abstract

Section: Image-based Methodsmentioning

confidence: 99%

Voxel- and Bird’s-Eye-View-Based Semantic Scene Completion for LiDAR Point Clouds

Liang,

Akhtar,

Vice

et al. 2024

Remote Sensing

View full text Add to dashboard Cite

show abstract

“…However, they produce predictions as sparse as the LiDAR point cloud, offering an incomplete understanding of the whole scene. Semantic scene completion [50] aims for dense inference of 3D geometry and semantics of objects and surfaces within a given extent, typically leveraging rich geometry information at the input extracted from depth [16,35], occupancy grids [58,49], point clouds [48], or a mix of modalities, e.g., RGBD [11,17]. In this line, MonoScene [12] is the first camera-based method to produce dense semantic occupancy predictions from a single image by projecting image features into 3D voxels by optical ray intersection.…”

Section: Related Workmentioning

confidence: 99%

Drive&Segment: Unsupervised Semantic Segmentation of Urban Scenes via Cross-Modal Distillation

Vobecký

Hurych

Siméoni

et al. 2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

We describe an approach to predict open-vocabulary 3D semantic voxel occupancy map from input 2D images with the objective of enabling 3D grounding, segmentation and retrieval of free-form language queries. This is a challenging problem because of the 2D-3D ambiguity and the open-vocabulary nature of the target tasks, where obtaining annotated training data in 3D is difficult. The contributions of this work are three-fold. First, we design a new model architecture for open-vocabulary 3D semantic occupancy prediction. The architecture consists of a 2D-3D encoder together with occupancy prediction and 3D-language heads. The output is a dense voxel map of 3D grounded language embeddings enabling a range of open-vocabulary tasks. Second, we develop a tri-modal self-supervised learning algorithm that leverages three modalities: (i) images, (ii) language and (iii) LiDAR point clouds, and enables training the proposed architecture using a strong pre-trained vision-language model without the need for any 3D manual language annotations. Finally, we demonstrate quantitatively the strengths of the proposed model on several open-vocabulary tasks: Zero-shot 3D semantic segmentation using existing datasets; 3D grounding and retrieval of free-form language queries, using a small dataset that we propose as an extension of nuScenes. You can find the project page here https://vobecant.github.io/POP3D.

show abstract

“…• As far as we know, our DOCTR is the first to introduce an object-centric Transformer-based network for the point scene understanding task that allows learning with multiple objects and multiple sub-tasks in a unified manner. Wu et al 2020;Yan et al 2021) often use voxelized input and then predict the semantic label of each voxel in both visible and occluded regions. They aim to jointly estimate the complete geometry and semantic labels from partial input.…”

Section: Introductionmentioning

confidence: 99%

DOCTR: Disentangled Object-Centric Transformer for Point Scene Understanding

Yu,

Wang,

et al. 2024

AAAI

View full text Add to dashboard Cite

Point scene understanding is a challenging task to process real-world scene point cloud, which aims at segmenting each object, estimating its pose, and reconstructing its mesh simultaneously. Recent state-of-the-art method first segments each object and then processes them independently with multiple stages for the different sub-tasks. This leads to a complex pipeline to optimize and makes it hard to leverage the relationship constraints between multiple objects. In this work, we propose a novel Disentangled Object-Centric TRansformer (DOCTR) that explores object-centric representation to facilitate learning with multiple objects for the multiple sub-tasks in a unified manner. Each object is represented as a query, and a Transformer decoder is adapted to iteratively optimize all the queries involving their relationship. In particular, we introduce a semantic-geometry disentangled query (SGDQ) design that enables the query features to attend separately to semantic information and geometric information relevant to the corresponding sub-tasks. A hybrid bipartite matching module is employed to well use the supervisions from all the sub-tasks during training. Qualitative and quantitative experimental results demonstrate that our method achieves state-of-the-art performance on the challenging ScanNet dataset. Code is available at https://github.com/SAITPublic/DOCTR.

show abstract

SCFusion: Real-time Incremental Scene Reconstruction with Semantic Completion

Cited by 22 publications

References 43 publications

Voxel- and Bird’s-Eye-View-Based Semantic Scene Completion for LiDAR Point Clouds

Voxel- and Bird’s-Eye-View-Based Semantic Scene Completion for LiDAR Point Clouds

Drive&Segment: Unsupervised Semantic Segmentation of Urban Scenes via Cross-Modal Distillation

DOCTR: Disentangled Object-Centric Transformer for Point Scene Understanding

Contact Info

Product

Resources

About