Total3DUnderstanding: Joint Layout, Object Pose and Mesh Reconstruction for Indoor Scenes From a Single Image

Abstract

Semantic reconstruction of indoor scenes refers to both scene understanding and object reconstruction. Existing works either address one part of this problem or focus on independent objects. In this paper, we bridge the gap between understanding and reconstruction, and propose an end-to-end solution to jointly reconstruct room layout, object bounding boxes and meshes from a single image. Instead of separately resolving scene understanding and object reconstruction, our method builds upon a holistic scene context and proposes a coarse-to-fine hierarchy with three components: 1. room layout with camera pose; 2. 3D object bounding boxes; 3. object meshes. We argue that understanding the context of each component can assist the task of parsing the others, which enables joint understanding and reconstruction. The experiments on the SUN RGB-D and Pix3D datasets demonstrate that our method consistently outperforms existing methods in indoor layout estimation, 3D object detection and mesh reconstruction.

Kimera: From SLAM to spatial perception with 3D dynamic scene graphs

“…Rogers and Christensen (2012) and Lin et al (2013) leveraged objects to perform a joint object-and-place classification. Nie et al (2020), Huang et al (2018a), and Zhao and Zhu (2013b) jointly solved the problem of scene understanding and reconstruction. Pangercic et al (2012) reasoned on the objects’ functionality.…”

Monte Carlo Scene Search for 3D Scene Understanding

“…Discriminative methods can exploit large training datasets to learn to classify scene components from input data such as RGB and RGB-D images [4,18,35,51,56]. By introducing clever Deep Learning architectures applied to point clouds or voxel-based representations, these methods can achieve very good results.…”

“…3D scene understanding is a fundamental problem in Computer Vision [41,53]. In the case of indoor scenes, one usually aims at recognizing the objects and their properties such as their 3D pose and geometry [2,3,15], or the room layouts [57,31,62,59,30,36,50,60,62,54,55], or both [4,18,35,45,51,56]. With the development of deep learning approaches, the field has made a remarkable progress.…”

An Improved Algorithm Robust to Illumination Variations for Reconstructing Point Cloud Models from Images

“…Since 2015, many deep-learning-based 3D reconstruction methods are presented, among which the point-based technique is simple but efficient in terms of memory requirements [36]. Similar to volumetric [37,38] and surface-based representations [39,40], point-based techniques follow the encoder-decoder model. In general, grid representations use up-convolutional networks to decode the latent variable [41,42].…”