Temporally Coherent General Dynamic Scene Reconstruction

Abstract

Existing techniques for dynamic scene reconstruction from multiple wide-baseline cameras primarily focus on reconstruction in controlled environments, with fixed calibrated cameras and strong prior constraints. This paper introduces a general approach to obtain a 4D representation of complex dynamic scenes from multi-view wide-baseline static or moving cameras without prior knowledge of the scene structure, appearance, or illumination. Contributions of the work are: an automatic method for initial coarse reconstruction to initialize joint estimation; sparse-to-dense temporal correspondence integrated with joint multi-view segmentation and reconstruction to introduce temporal coherence; and a general robust approach for joint segmentation refinement and dense reconstruction of dynamic scenes by introducing shape constraint. Comparison with state-of-the-art approaches on a variety of complex indoor and outdoor scenes, demonstrates improved accuracy in both multiview segmentation and dense reconstruction. This paper demonstrates unsupervised reconstruction of complete temporally coherent 4D scene models with improved non-rigid object segmentation and shape reconstruction and its application to various applications such as free-view rendering and virtual reality.

The One Where They Reconstructed 3D Humans and Environments in TV Shows

“…Reconstruction of 3D scenes is a well studied problem, e.g., [1,21,53,54], however, most methods assume static scenes. Related work focuses on dynamic reconstruction [4,38], but requires capture from multiple widebaseline synchronized cameras. Luo et al [33] and Kopf et al [29] present pipelines for recovering depth in monocular videos that include humans, but they assume that the underlying scene is static.…”

Neural Free-Viewpoint Performance Rendering under Complex Human-object Interactions

“…All these methods suffer from the limited mesh resolution leading to uncanny texturing output. Recent method [37] leverages unsupervised temporally coherent human reconstruction to generate free-viewpoint rendering. It is still hard for this method to get photo-realistic rendering results.…”

Line Laser Scanning Combined with Machine Learning for Fish Head Cutting Position Identification

“…Dynamic 3D reconstruction is one of the important technologies for fast and accurate contour feature identification in advanced manufacturing, and food and agricultural product processing [4,5]. The acquisition of the measured object contour data is mainly realized by means of computer vision, lasers, ultrasound, NMR, and X-rays.…”