Real-time sequential model-based non-rigid SFM

Abstract

Tracking non-rigid objects from video is useful in robotic systems such as HMIs or robotic manipulator arms which interact with deformable objects. This paper proposes a method for sequential model-based 3D reconstruction of deformable objects and camera localization in real time. Nonrigid SFM methods commonly process a video sequence offline in a batch way. While there are real-time methods for rigid models, reconstruction of deformable 3D shapes for real-time applications is still unsolved. Dense approaches offer promising results, but processing all frames in batch, offline. We propose a real-time non-rigid reconstruction method based on a known deformable model. Object shape and pose is tracked by realtime estimation of camera pose and deformation coefficients. An extensive evaluation of the algorithm on several data sets, and comparison with state-of-the-art techniques is performed. The tests include different outlier rates, noise levels and occlusions handling.

Model-Based Real-Time Non-Rigid Tracking

“…Our solution deals with data association in real time. This paper extends the work published by the authors in [ 52 ] where a preliminary version of our tracking algorithm was presented. In this work, an improved version is implemented, and a deeper analysis is carried out.…”

“…Two different feature detection and matching methods are studied. The first one is based on PTAM, which uses the FAST detector and was presented by the authors in [ 52 ]. This is considered as the baseline for the comparison.…”

“…In case the correlation is too low or the distance is too high, the matching is considered not found. For a deeper explanation, we refer the readers to our publication in [ 52 ].…”

Model-free non-rigid head pose tracking by joint shape and pose estimation

“…Building upon the rigid factorisation algorithm in [32], the proposed batch-processing method employed SVD to decompose a measurement matrix of higher rank into pose, basis shapes and their corresponding configuration weights. The formulation of the shape model proposed in [10] was adopted significantly by subsequent batch-processing [8,33,40,41] and sequential [1,5,11,27] approaches for non-rigid shape and motion recovery. A shortcoming of this model, however, is its sensitivity to the number of basis shapes that define the degrees-of-freedom of the surface deformations, where a restrictive model with too few basis shapes fails to model the measurement data well, while too many basis shapes in the model erroneously capture the noise in the data.…”

“…Methods which operate under the assumption of non-degeneracy introduce additional degreesof-freedom that, in the presence of degenerate deformations, are usually not descriptive of the measurement data and would erroneously capture the noise [1,5,8,10,11,33,40]. …”