Multiview 3D Tracking with an Incrementally Constructed 3D Model

Abstract

We propose a multiview tracking method for rigid objects. Assuming that a part of the object is visible in at least two cameras, a partial 3D model is reconstructed in terms of a collection of small 3D planar patches of arbitrary topology. The 3D representation, recovered fully automatically, allows to formulate tracking as gradient minimization in pose (translation, rotation) space. As the object moves, the 3D model is incrementally updated. A virtuous circle emerges: tracking enables composition of the partial 3D model; the 3D model facilitates and robustifies the multiview tracking.We demonstrate experimentally that the interleaved track-and-reconstruct approach successfully tracks a 360 degrees turn-around and a wide range of motions. Monocular tracking is also possible after the model is constructed. Using more cameras, however, significantly increases stability in critical poses and moves. We demonstrate how to exploit the 3D model to increases stability in the presence of uneven and/or changing illumination.

Scene flow constrained multi-prior patch-sweeping for real-time upper body 3D reconstruction

“…Therefore, its first order approximation, namely an element of the group of 3 × 3 skew-symmetric matrices (the Lie group of infinitesimal 3D rotations), is considered sufficient for optimization. This approximation enables a more powerful Newton-Raphson numerical scheme proposed by Zimmermann et al [10].…”

Multi-object detection and tracking by stereo vision

“…Still, this system cannot detect an occluded head in tracking. Recently, a new tracking and reconstructing method [34] for the single rigid object is introduced to build the 3D model incrementally by one stereo pair and then estimate the motion parameters by a single camera. Instead of tracking independent objets in 3D space, Mozerov et al [35] present an action-specific model of human motion to track the correlated body parts by automatically synchronizing the training sequences and learning 3D human postures.…”

“…In addition, these systems usually require additional modules (e.g. face or skin detection [31], enhancement [32], scale filtering [33] or 3D model [34]) to reduce the depth noise.…”

Multi-camera 3D scene reconstruction from vanishing points

“…Examples of vision systems using multiple vision sensors include the Vi Room which is a low cost synchronized multicamera system manufactured by [1],the UD Multi Camera Project proposed by [2], the Cognitive Multi-Camera System developed by [3] and the real-time 3D Tele-immersion cluster system developed by [4]. These systems are able to capture multiple synchronized images of indoor scenes and they were developed for tracking applications without providing any knowledge for the arrangement of the 3D environment or for light field applications where information of the 3D volumetric structure of the scene is not necessary.…”