Joint Multi-Layer Segmentation and Reconstruction for Free-Viewpoint Video Applications

Abstract

Current state-of-the-art image-based scene reconstruction techniques are capable of generating high-fidelity 3D models when used under controlled capture conditions. However, they are often inadequate when used in more challenging environments such as sports scenes with moving cameras. Algorithms must be able to cope with relatively large calibration and segmentation errors as well as input images separated by a wide-baseline and possibly captured at different resolutions. In this paper, we propose a technique which, under these challenging conditions, is able to efficiently compute a high-quality scene representation via graph-cut optimisation of an energy function combining multiple image cues with strong priors. Robustness is achieved by jointly optimising scene segmentation and multiple view reconstruction in a view-dependent manner with respect to each input camera. Joint optimisation prevents propagation of errors from segmentation to reconstruction as is often the case with sequential approaches. View-dependent processing increases tolerance to errors in through-the-lens calibration compared to global approaches. We evaluate our technique in the case of challenging outdoor sports scenes captured with manually operated broadcast cameras as well as several indoor scenes with natural background. A comprehensive experimental evaluation including qualitative and quantitative results demonstrates the accuracy of the technique for high quality segmentation and reconstruction and its suitability for free-viewpoint video under these difficult conditions.Keywords Segmentation · Reconstruction · Free-viewpoint video · Graph-cuts Centre for Vision, Speech and Signal Processing University of Surrey, Guildford, GU2 7XH, UK Tel.: +44 1483 683958 Fax: +44 1483 686031 E-mail: J.Guillemaut@surrey.ac.uk Fig. 1 Two moving broadcast camera views (top) and a locked-off camera view (bottom-left) from a rugby match.

A Saliency-based Framework for 2D-3D Registration

“…The two real datasets are the dinosaur and temple from Middlebury's multi-view reconstruction dataset [Seitz et al, 2006]. Since this does not include the ground truth, we use the 3D reconstruction provided by [Guillemaut and Hilton, 2011]. These images contain texture information of the model while the 3D data does not, adding another layer of difficulty.…”

Space-Time Joint Multi-layer Segmentation and Depth Estimation

“…While a few methods have been previously proposed for joint segmentation and reconstruction [12,32,15,14], these all performed frame-by-frame processing of the input data and therefore lacked temporal consistency. The proposed method is the first joint segmentation and reconstruction approach which explicitly enforces temporal consistency.…”

“…To make the generalisation to the temporal domain more evident, we adopt a similar formalism as in [15,14] which defined a single frame layered depth estimation energy minimisation framework.…”

“…Application of this algorithm requires the energy to be regular [17]. After observing that the spatial and temporal terms have a similar structure differing only in terms of coefficients independent of the labels, proof of the regularity follows immediately from the known regularity of the spatial component of the energy (see [14] for a proof in that case). As previously observed, a key challenge in extending layered depth estimation to the temporal domain is scalability.…”

Intrinsic Textures for Relightable Free-Viewpoint Video

“…FVVR has been applied successfully in sports TV production [4,5] and video conferencing [6] among other applications. In FVVR, video from several cameras is used to reconstruct scene geometry using Multiple-View Stereo (MVS), and appearance is reproduced by projectively texturing the scene with the original images [7].…”