Im2Pano3D: Extrapolating 360° Structure and Semantics Beyond the Field of View

Song, Shuran; Zeng, Andy; Chang, Angel X.; Savva, Manolis; Savarese, Silvio; Funkhouser, Thomas

doi:10.1109/cvpr.2018.00405

Cited by 77 publications

(58 citation statements)

References 40 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…distance-to-origin). We then pass both predicted outputs through a differentiable PN-layer [22] to convert the estimated surface normals and plane distances into a pixel-wise prediction of 3D locations. Direct supervision is provided to the 1) surface normal predictions via a cosine loss, 2) plane offset predictions via an 1 loss, and 3) final 3D point locations via an 1 to ensure consistency between the surface normal and plane offset predictions.…”

Section: Geometry Estimationmentioning

confidence: 99%

Neural Illumination: Lighting Prediction for Indoor Environments

Song¹,

Funkhouser

2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

Self Cite

111

149

View full text Add to dashboard Cite

This paper addresses the task of estimating the light arriving from all directions to a 3D point observed at a selected pixel in an RGB image. This task is challenging because it requires predicting a mapping from a partial scene observation by a camera to a complete illumination map for a selected position, which depends on the 3D location of the selection, the distribution of unobserved light sources, the occlusions caused by scene geometry, etc. Previous methods attempt to learn this complex mapping directly using a single black-box neural network, which often fails to estimate high-frequency lighting details for scenes with complicated 3D geometry. Instead, we propose "Neural Illumination," a new approach that decomposes illumination prediction into several simpler differentiable sub-tasks: 1) geometry estimation, 2) scene completion, and 3) LDR-to-HDR estimation. The advantage of this approach is that the subtasks are relatively easy to learn and can be trained with direct supervision, while the whole pipeline is fully differentiable and can be fine-tuned with end-to-end supervision. Experiments show that our approach performs significantly better quantitatively and qualitatively than prior work.

show abstract

Section: Geometry Estimationmentioning

confidence: 99%

Neural Illumination: Lighting Prediction for Indoor Environments

Song¹,

Funkhouser

2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

Self Cite

111

149

View full text Add to dashboard Cite

show abstract

“…. Following the convention [37,31], we formulate the input to both scan completion modules using a similar tensor form…”

Section: Scan Completion Modulesmentioning

confidence: 99%

Extreme Relative Pose Estimation for RGB-D Scans via Scene Completion

Yang

Pan

Luo

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

View full text Add to dashboard Cite

Estimating the relative rigid pose between two RGB-D scans of the same underlying environment is a fundamental problem in computer vision, robotics, and computer graphics. Most existing approaches allow only limited maximum relative pose changes since they require considerable overlap between the input scans. We introduce a novel deep neural network that extends the scope to extreme relative poses, with little or even no overlap between the input scans. The key idea is to infer more complete scene information about the underlying environment and match on the completed scans. In particular, instead of only performing scene completion from each individual scan, our approach alternates between relative pose estimation and scene completion. This allows us to perform scene completion by utilizing information from both input scans at late iterations, resulting in better results for both scene completion and relative pose estimation. Experimental results on benchmark datasets show that our approach leads to considerable improvements over state-of-the-art approaches for relative pose estimation. In particular, our approach provides encouraging relative pose estimates even between non-overlapping scans.

show abstract

“…Flint et al used a combination of monocular features, with multiple-view and 3D features to infer a Manhattan World representation of the environment [9]. In [30], a RGB-D panorama is split and half of the scene is taken as input with the aim of producing reasonable, cluttered room layout estimation for the unseen portion of the panorama.…”

Section: Cloudmentioning

confidence: 99%

Thinking Outside the Box: Generation of Unconstrained 3D Room Layouts

Howard-Jenkins

Prisacariu

2019

Computer Vision – ACCV 2018

View full text Add to dashboard Cite

We propose a method for room layout estimation that does not rely on the typical box approximation or Manhattan world assumption. Instead, we reformulate the geometry inference problem as an instance detection task, which we solve by directly regressing 3D planes using an R-CNN. We then use a variant of probabilistic clustering to combine the 3D planes regressed at each frame in a video sequence, with their respective camera poses, into a single global 3D room layout estimate. Finally, we showcase results which make no assumptions about perpendicular alignment, so can deal effectively with walls in any alignment.

show abstract

Im2Pano3D: Extrapolating 360° Structure and Semantics Beyond the Field of View

Cited by 77 publications

References 40 publications

Neural Illumination: Lighting Prediction for Indoor Environments

Neural Illumination: Lighting Prediction for Indoor Environments

Extreme Relative Pose Estimation for RGB-D Scans via Scene Completion

Thinking Outside the Box: Generation of Unconstrained 3D Room Layouts

Contact Info

Product

Resources

About