Free-viewpoint video of human actors

Abstract

In free-viewpoint video, the viewer can interactively choose his viewpoint in 3-D space to observe the action of a dynamic realworld scene from arbitrary perspectives. The human body and its motion plays a central role in most visual media and its structure can be exploited for robust motion estimation and efficient visualization. This paper describes a system that uses multi-view synchronized video footage of an actor's performance to estimate motion parameters and to interactively re-render the actor's appearance from any viewpoint. The actor's silhouettes are extracted from synchronized video frames via background segmentation and then used to determine a sequence of poses for a 3D human body model. By employing multi-view texturing during rendering, time-dependent changes in the body surface are reproduced in high detail. The motion capture subsystem runs offline, is non-intrusive, yields robust motion parameter estimates, and can cope with a broad range of motion. The rendering subsystem runs at real-time frame rates using ubiquous graphics hardware, yielding a highly naturalistic impression of the actor. The actor can be placed in virtual environments to create composite dynamic scenes. Free-viewpoint video allows the creation of camera fly-throughs or viewing the action interactively from arbitrary perspectives.

Shape-From-Silhouette Across Time Part II: Applications to Human Modeling and Markerless Motion Tracking

“…Among all of the model based approaches to track human motion, the work by Sidenbladh et al in (2000bSidenbladh et al in ( , 2000a, that by Delamarre and Faugeras in (1999), that by Carranza et al in (2003) and that by Mikic et al in (2003) are the most related to our tracking algorithm. Sidenbladh et al (2000a) perform human motion tracking by first modeling the person using articulated cylinders as body parts.…”

“…The way of generating "forces" to align 2D contours of the projected model with the silhouette boundary is similar to the geometric constraints we use in our tracking algorithm. In Carranza et al (2003), first render a human model using graphics hardware and then compare the rendered images (using pixel-wise XOR) with the silhouette images extracted from video sequences to track human motion. Although it is unclear exactly how their XOR errors are formulated as driving forces for optimizing the motion parameters, their gridsearch initialization procedure provides a good way to reduce the problem of local minima.…”

“…Moreover there are also a large number of systems for tracking human motion in video sequences (Rehg and Kanade, 1995;Gavrila and Davis, 1996;Malik, 1997, 1998;Yamamoto et al, 1998;Haritaoglu et al, 1998;Jojic et al, 1999;DiFranco et al, 1999;Rehg, 1999a, 1999b;Pavlovic et al, 1999;Delamarre and Faugeras, 1999;Cheung et al, 2000;Sidenbladh et al, 2000aSidenbladh et al, , 2000bDeutscher et al, 2000;Difranco et al, 2001;Drummond and Cipolla, 2001;Liebowitz and Carlsson, 2001;Sullivan and Carlsson, 2002;Mikic et al, 2003;Carranza et al, 2003) using a variety of model-based approaches. An extensive survey of vision-based motion tracking systems can be found in Moeslund and Granum (2001).…”

“…An extensive survey of vision-based motion tracking systems can be found in Moeslund and Granum (2001). Among the above systems, silhouette information has been used extensively (Cai and Aggarwal, 1996;Wren et al, 1997;Kakadiaris and Metaxas, 1998;Cai and Aggarwal, 1998;Beymer and Konolige, 1999;Cheung et al, 2000;Mikic et al, 2003;Carranza et al, 2003) since silhouettes are easy to extract and provide valuable information about the position and shape (posture) of the person. In particular, many human shape modeling and motion tracking systems (such as (Moezzi et al, 1997;Kakadiaris and Metaxas, 1998) and more recently (Cheung et al, 2000;Matusik, 2001;Mikic et al, 2001)) use ShapeFrom-Silhouette to construct 3D estimates of the body shape for modeling and tracking.…”

“…rectangles or ellipses in 2D, cylinders or ellipsoids in 3D) to model the body parts of the person. Finally our tracking algorithm incorporates both silhouette and color information at the same time instead of using only one of the two cues (Delamarre and Faugeras, 1999;Sidenbladh et al, 2000bSidenbladh et al, , 2000aCarranza et al, 2003).…”