Real-Time Dynamic 3-D Object Shape Reconstruction and High-Fidelity Texture Mapping for 3-D Video

Abstract

Abstract-3D video is a real 3D movie recording the object's full 3D shape, motion, and precise surface texture. This paper first proposes a parallel pipeline processing method for reconstructing dynamic 3D object shape from multi-view video images, by which a temporal series of full 3D voxel representations of the object behavior can be obtained in real-time. To realize the real-time processing, we first introduce a plane-based volume intersection algorithm: represent an observable 3D space by a group of parallel plane slices, back-project observed multi-view object silhouettes onto each slice, and apply 2D silhouette intersection on each slice. Then, we propose a method to parallelize this algorithm using a PC cluster, where we employ 5 stage pipeline processing in each PC as well as slice-by-slice parallel silhouette intersection. Several results of the quantitative performance evaluation are given to demonstrate the effectiveness of the proposed methods. In the latter half of the paper, we present an algorithm of generating video texture on the reconstructed dynamic 3D object surface. We first describe a naive viewindependent rendering method and show its problems. Then, we improve the method by introducing image-based rendering techniques. Experimental results demonstrate the effectiveness of the improved method in generating high fidelity object images from arbitrary viewpoints.

Refinement of Visual Hulls for Human Performance Capture

“…For higher-quality modeling, especially for reconstructing concave parts, sophisticated model refinement algorithms are required after the shape-from-silhouette, such as deformable mesh (Matsuyama et al, 2004), stereo matching, (Starck & Hilton, 2007;Tomiyama et al, 2004) and graph cuts in the 3D space (Hisatomi et al, 2008;Tung et al, 2008). However, such a model refinement process is out of the scope of this chapter.…”

“…Generation of dynamic three-dimensional (3D) mesh sequences of human performance using multiple cameras has been actively investigated in recent years (de Aguiar et al, 2008;Hisatomi et al, 2008;Kanade et al, 1997;Kim et al, 2007;Matsuyama et al, 2004;Nobuhara & Matsuyama, 2003;Snow et al, 2000;Starck & Hilton, 2007;Tomiyama et al, 2004;Toyoura et al, 2007;Tung et al, 2008;Vlasic et al, 2008). The topic is drawing a lot of attention because conventional 3D shape measurement tools, such as laser scanners, shape (structure)-from-motion (Huang & Netravali, 1994;Poelman & Kanade, 1997), shape-from-shading (Zhang et al, 1999), etc., are difficult to apply to dynamic scenes.…”

Ubiquitous and Wearable Vision Systems

“…To make full use of the captured video data, we developed a system for generating 3D video [4][9] [10].…”

“…We proposed the viewpoint dependent vertex-based texture mapping method to avoid jitters in rendered object images which are caused due to the limited accuracy of the reconstructed 3D object shape [4]. Experimental results showed that the proposed method can generate almost natural looking object images from arbitrary viewpoints.…”

“…We may call such system a ubiquitous vision system. Based on this scheme, we developed a cooperative distributed active object tracking system [2] and a 3D video [3] generation system [4]. Experimental results demonstrated utilities of the ubiquitous vision system.…”