Temporally Coherent 4D Reconstruction of Complex Dynamic Scenes

Abstract

This paper presents an approach for reconstruction of 4D temporally coherent models of complex dynamic scenes. No prior knowledge is required of scene structure or camera calibration allowing reconstruction from multiple moving cameras. Sparse-to-dense temporal correspondence is integrated with joint multi-view segmentation and reconstruction to obtain a complete 4D representation of static and dynamic objects. Temporal coherence is exploited to overcome visual ambiguities resulting in improved reconstruction of complex scenes. Robust joint segmentation and reconstruction of dynamic objects is achieved by introducing a geodesic star convexity constraint. Comparative evaluation is performed on a variety of unstructured indoor and outdoor dynamic scenes with hand-held cameras and multiple people. This demonstrates reconstruction of complete temporally coherent 4D scene models with improved nonrigid object segmentation and shape reconstruction.

Semantically Coherent Co-Segmentation and Reconstruction of Dynamic Scenes

“…To improve the results joint optimisation of segmentation with 3D reconstruction has been proposed [21,42] by including the multiple view photo-consistency. This concept was extended to semantic segmentation and reconstruction to obtain additional information from the scene [24,56].…”

“…O k is the set of k most photo-consistent pairs with reference camera. Smoothness cost: The surface smoothness cost introduced in [42] is extended to spatial and temporal neighbourhoods: The importance of the proposed semantically coherent optimization exploiting the information from semantic labels and tracklets for single and multiple views is shown in the Figure 5. Comparison is presented against optimization with/without semantic label and temporal tracklet information for single and multiple views.…”

“…Previous research has demonstrated the advantages of joint segmentation and reconstruction across multiple views [21,24,23,34,14,32], co-segmentation of multiple view images [11,31,13,12] and temporal coherence in reconstruction [20,18,36,42]. Our contribution is the introduction of a framework for joint semantic co-segmentation and reconstruction of complex dynamic scenes to obtain semantically coherent per-view 2D object segmentation and 3D scene reconstruction from wide-baseline camera views.…”

4D Match Trees for Non-rigid Surface Alignment

“…The input is a partial surface reconstruction or depth map of a general dynamic scenes at each frame together with single or multiple view images. Cameras may be static or moving and camera calibration is assumed to be known or estimated together with the scene reconstruction [31,32,3,20]. The first step is to estimate sparse wide-timeframe feature correspondence.…”

“…Consistent mesh sequences finds application in performance capture, animation and motion analysis. A number of approaches for surface reconstruction [19,20] do not produce temporally coherent models for an entire sequence rather they align pairs of frames sequentially. Other methods proposed for 4D alignment of surface reconstructions assume that a complete mesh of the dynamic object is available for the entire sequence [21,22,23,24,25].…”