Self-supervised Single-View 3D Reconstruction via Semantic Consistency

Li, Xueting; Liu, Sifei; Kim, Kihwan; Mello, Shalini De; Jampani, Varun; Yang, Ming–Hsuan; Kautz, Jan

doi:10.1007/978-3-030-58568-6_40

Cited by 89 publications

(126 citation statements)

References 30 publications

Supporting

Mentioning

126

Contrasting

Order By: Relevance

“…Since this loss is discontinuous, they use the derivative-free Nelder-Mead optimisation method. In very recent, ambitious work, Li et al [18] learn both a deformable model and model fitting in a self-supervised fashion. One of their training objectives is to ensure semantic consistency, measured by projecting the semantically labelled 3D model into the image.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Shape from semantic segmentation via the geometric Rényi divergence

Koizumi

Smith

2021

2021 IEEE Winter Conference on Applications of Computer Vision (WACV)

View full text Add to dashboard Cite

show abstract

Section: Related Workmentioning

confidence: 99%

“…Meanwhile, SoftRas compares a soft rasterisation to hard discrete input meaning that the minimum loss does not correspond to optimal alignment. No previous work, including [32,7,18], has considered the problem of estimating shape using only semantic segmentation information.…”

Section: Related Workmentioning

confidence: 99%

Shape from semantic segmentation via the geometric Rényi divergence

Koizumi

Smith

2021

2021 IEEE Winter Conference on Applications of Computer Vision (WACV)

View full text Add to dashboard Cite

show abstract

“…When ground-truth 3D models are not available, dynamic neural radiance fields (NeRF) [54,43,63,61] learn an implicit scene representation from videos, but are often scene-specific and don't scale to a class-agnostic setting. Self-supervised methods [91,35,41] are promising and learn 4D reconstruction via 2D supervision of differentiable rendering [37,48]. In contrast, in this work, we present a class-agnostic and template-free framework which learns to recover the shape and dynamics from input videos.…”

Section: Geometric Representationsmentioning

confidence: 99%

Class-agnostic Reconstruction of Dynamic Objects from Videos

Ren¹,

Zhao²,

Schwing³

2021

Preprint

View full text Add to dashboard Cite

We introduce REDO, a class-agnostic framework to REconstruct the Dynamic Objects from RGBD or calibrated videos. Compared to prior work, our problem setting is more realistic yet more challenging for three reasons: 1) due to occlusion or camera settings an object of interest may never be entirely visible, but we aim to reconstruct the complete shape; 2) we aim to handle different object dynamics including rigid motion, non-rigid motion, and articulation; 3) we aim to reconstruct different categories of objects with one unified framework. To address these challenges, we develop two novel modules. First, we introduce a canonical 4D implicit function which is pixel-aligned with aggregated temporal visual cues. Second, we develop a 4D transformation module which captures object dynamics to support temporal propagation and aggregation. We study the efficacy of REDO in extensive experiments on synthetic RGBD video datasets SAIL-VOS 3D and DeformingThings4D++, and on real-world video data 3DPW. We find REDO outperforms state-of-the-art dynamic reconstruction methods by a margin. In ablation studies we validate each developed component.

show abstract

“…An automatic but robust autocalibration method for multiview human performance capture [4] is urgent. Examples include 3D human reconstruction from sparse views without pose parameters [5], selfcalibration [6] for systems when there are disturbances to camera setups [7], and enhancing the reconstruction performance due to the inaccurate camera calibration [8].…”

Section: Introductionmentioning

confidence: 99%

Relative Pose Estimation for RGB-D Human Input Scans via Implicit Function Reconstruction

Liu

Zhang

et al. 2022

Wireless Communications and Mobile Computing

View full text Add to dashboard Cite

To achieve a promising performance on relative pose estimation for RGB-D scans, a considerable overlap between two RGB-D inputs is often required for most existing methods. However, in many practical applications for human scans, we often have to estimate the relative poses under arbitrary overlaps, which is challenging for existing methods. To deal with this problem, this paper presents a novel end-to-end and coarse-to-fine optimization method. Our method is self-supervision which firstly combines implicit function reconstruction with differentiable render for RGB-D human input scans at arbitrary overlaps in relative pose estimation. The insight is to take advantage of the underlying human geometry prior as much as possible. First of all, for stable coarse poses, we utilize the implicit function reconstruction to dig out abundant hidden cues from unseen regions in the initialization module. To further refine the poses, the differentiable render is leveraged to establish a self-supervision mechanism in the optimization module, which is independent of standard pipelines for feature extracting and accurate correspondence matching. More importantly, our proposed method is flexible to be extended to multiview input scans. The results and evaluations demonstrate that our optimization module is robust for real-world noisy inputs, and our approach outperforms considerably than standard pipelines in non-overlapping setups.

show abstract

Self-supervised Single-View 3D Reconstruction via Semantic Consistency

Cited by 89 publications

References 30 publications

Shape from semantic segmentation via the geometric Rényi divergence

Shape from semantic segmentation via the geometric Rényi divergence

Class-agnostic Reconstruction of Dynamic Objects from Videos

Relative Pose Estimation for RGB-D Human Input Scans via Implicit Function Reconstruction

Contact Info

Product

Resources

About