Joint motion and reflectance capture for relightable 3D video

Theobalt, Christian; Ahmed, Naveed; Aguiar, Edilson de; Ziegler, Gernot; Lensch, Hendrik P. A.; Magnor, Marcus; Seidel, Hans‐Peter

doi:10.1145/1187112.1187200

Cited by 9 publications

(9 citation statements)

References 5 publications

(4 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In Section 7, we briefly talk about an extension of our original model-based free-viewpoint video pipeline that enables us to capture shape, motion parameters and dynamic surface reflectance of the whole human body at high accuracy [41,47]. As input, we only expect a handful of synchronized video streams showing arbitrary human motion.…”

Section: Image-based Reflectance Estimationmentioning

confidence: 99%

“…Thus, we can fit a dynamic surface reflectance model to each point on the body surface which consists of a per-texel parametric bidirectional reflectance distribution function and a per-texel time-varying normal direction. We now briefly review the algorithmic steps needed to generate relightable freeviewpoint videos and refer the interested reader to [41,47] for technical details.…”

Section: Relightable Free-viewpoint Videomentioning

confidence: 99%

“…The motion of textiles is identified and compensated by optical flow computation and texture warping. Please refer to [41,47] for the details of these methods.…”

Section: Modifications To Original Pipelinementioning

confidence: 99%

See 2 more Smart Citations

Video-based Capturing and Rendering of People

2008

Self Cite

View full text Add to dashboard Cite

Abstract. The interesting combination of Chapters in this book gives a nice overview of the importance of the analysis of human motion in three different scientific disciplines, computer vision, biomechanics, and computer graphics. In the computer vision field the main goal has been to develop robust and efficient techniques for capturing and measuring human motion. Researchers from biomechanics, on the other hand, try to capitalize on such measurement methods to analyze the characteristics of human motion. In contrast, the focus of computer graphics research has been to realistically animate and render moving humans. In this Chapter, we will outline that a joint solution to the acquisition, the reconstruction, and the rendering problem paves the trail for video-based capturing and rendering of people. We present a model-based system to create realistic 3D videos of human actors that puts this idea into practice. Our method enables us to capture human actors with multiple video cameras and to render their performances in real-time from arbitrary novel viewpoints and under arbitrary novel lighting conditions.

show abstract

Section: Image-based Reflectance Estimationmentioning

confidence: 99%

Section: Relightable Free-viewpoint Videomentioning

confidence: 99%

See 1 more Smart Citation

Video-based Capturing and Rendering of People

2008

Self Cite

View full text Add to dashboard Cite

show abstract

“…One common editing task for seamless cut & paste of objects into new environments would be relighting. However, as shown by Theobalt et al [21] extracting reflectance properties from time-varying multi-view video data is a challenging task and no solution for high-quality re-rendering has been proposed so far.…”

Section: Related Workmentioning

confidence: 99%

Interactive 3D video editing

2006

View full text Add to dashboard Cite

We present a generic and versatile framework for interactive editing of 3D video footage. Our framework combines the advantages of conventional 2D video editing with the power of more advanced, depth-enhanced 3D video streams. Our editor takes 3D video as input and writes both 2D or 3D video formats as output. Its underlying core data structure is a novel 4D spatio-temporal representation which we call the video hypervolume. Conceptually, the processing loop comprises three fundamental operators: slicing, selection, and editing. The slicing operator allows users to visualize arbitrary hyperslices from the 4D data set. The selection operator labels subsets of the footage for spatio-temporal editing. This operator includes a 4D graph-cut based algorithm for object selection.The actual editing operators include cut & paste, affine transformations, and compositing with other media, such as images and 2D video. For high-quality rendering, we employ EWA splatting with view-dependent texturing and boundary matting. We demonstrate the applicability of our methods to post-production of 3D video.

show abstract

“…[Wilburn et al 2005], [Wenger et al 2005]. [Theobalt et al 2005] addressed control both of the viewpoint and illumination, however the approach is challenge by low sampling of both lighting and view dimensions. We take a step toward an image-based approach to obtaining postproduction control over both viewpoint and illumination of cyclic full-body human motion by combining the performance relighting technique of [Wenger et al 2005] with a novel view generation technique based on a flowed reflectance field.…”

mentioning

confidence: 99%

Relighting human locomotion with flowed reflectance fields

Chabert

Einarsson

Jones

et al. 2006

ACM SIGGRAPH 2006 Sketches on - SIGGRAPH '06

View full text Add to dashboard Cite

Overview We present an image-based approach for capturing the appearance of a walking or running person so they can be rendered realistically under variable viewpoint and illumination. Considerable work has addressed aspects of postproduction control of viewpoint and illumination of a human performance. Most proposed systems address only one of those two aspects e.g. [Wilburn et al. 2005], [Wenger et al. 2005]. [Theobalt et al. 2005] addressed control both of the viewpoint and illumination, however the approach is challenge by low sampling of both lighting and view dimensions. We take a step toward an image-based approach to obtaining postproduction control over both viewpoint and illumination of cyclic full-body human motion by combining the performance relighting technique of [Wenger et al. 2005] with a novel view generation technique based on a flowed reflectance field. By restricting our consideration to cyclic motion such as walking and running, we are able to acquire a 2D array of views by slowly rotating the subject in front of a 1D vertical array of three high speed cameras and segmenting the data per motion cycle. We then use a combination of light field rendering and view interpolation based on optical flow to render the subject from new viewpoints. Capture The data were captured using an 8m lighting apparatus related to [Wenger et al. 2005] but designed for full human body capture ( Fig. 1(a)). We programmed the device with a 33-frame lighting sequence with 26 basis lighting conditions, three evenlyspaced tracking frames, three corresponding matte frames and a stripe frame (not used at present). We capture the subject using a high speed camera with the lighting basis running at 30 fps/s that give us a a total frequency of 990Hz. . At this frame rate, we can capture 36 motion cycles at 320x448 pixels within the 8 GB memory capacity of the high speed camera. For each motion cycle we have 32 and 25 lighting sequences for walking and running data sets respectively. The 36 locomotion cycles recorded by the three cameras,if assumed identical, yield 108 relightable views of the walk cylcle. * e-mail: {last name}@ict.usc.eduRegistration Once the data is captured we first compute the alpha channels and pre-matte the dataset; this increases the compression ratio and also helps the optical flow algorithm. We then compute optical flow to spatially register the dataset, similar to [Wilburn et al. 2005]. The images are temporally registered in each lighting sequence using a process similar to [Wenger et al. 2005]. The registration is performed by warping all the frames within a lighting sequence toward the tracking frame in the middle of the sequence. After registration we have the equivalent of a 36×3 grid ( Fig. 1 (b)) of 4D reflectance fields. To create the flowed reflectance field, we compute flow fields between each viewpoint and its 4-neighbors.Rendering Our rendering process consists of five steps: lighting, image warping, light field interpolation, shadow rendering, and compositing. We first relight the ref...

show abstract

Joint motion and reflectance capture for relightable 3D video

Abstract: a) (b) Figure 1: (a) 3D videos rendered under different novel lighting conditions. (b) Wrinkles in the trousers are captured in the normal field.

Cited by 9 publications

References 5 publications

Video-based Capturing and Rendering of People

Video-based Capturing and Rendering of People

Interactive 3D video editing

Relighting human locomotion with flowed reflectance fields

Contact Info

Product

Resources

About