Articulated distance fields for ultra-fast tracking of hands interacting

Abstract

The state of the art in articulated hand tracking has been greatly advanced by hybrid methods that fit a generative hand model to depth data, leveraging both temporally and discriminatively predicted starting poses. In this paradigm, the generative model is used to define an energy function and a local iterative optimization is performed from these starting poses in order to find a "good local minimum" (i.e. a local minimum close to the true pose). Performing this optimization quickly is key to exploring more starting poses, performing more iterations and, crucially, exploiting high frame rates that ensure that temporally predicted starting poses are in the basin of convergence of a good local minimum. At the same time, a detailed and accurate generative model tends to deepen the good local minima and widen their basins of convergence. Recent work, however, has largely had to trade-off such a detailed hand model with one that facilitates such rapid optimization. We present a new implicit model of hand geometry that mostly avoids this compromise and leverage it to build an ultra-fast hybrid hand tracking system. Specifically, we construct an articulated signed distance function that, for any pose, yields a closed form calculation of both the distance to the detailed surface geometry and the necessary derivatives to perform gradient based optimization. There is no need to introduce or update any explicit "correspondences" yielding a simple algorithm that maps well to parallel hardware such as GPUs. As a result, our system can run at extremely high frame rates (e.g. up to 1000fps). Furthermore, we demonstrate how to detect, segment and optimize for two strongly interacting hands, recovering complex interactions at extremely high framerates. In the absence of publicly available datasets of sufficiently high frame rate, we leverage a multiview capture system to create a new 180fps dataset of one and two hands interacting together or with objects.

Resolving 3D Human Pose Ambiguities With 3D Scene Constraints

“…By knowing the scene structure we could reason about what is visible and what is not. Another interesting direction would be the unification of the self-penetration and the body-scene interpenetration by employing the implicit formulation of [65] for the whole body. Future work can exploit recent deep networks to estimate the scene directly from monocular RGB images.…”

Online optical marker-based hand tracking with deep labels

“…There is a rich body of work on markerless tracking of hand motion from either depth [Oikonomidis et al 2012;Taylor et al 2016;Tkach et al 2016;Tompson et al 2014] or RGB cameras. Although high-end 3D scanners or depth cameras can generate high-fidelity geometry at high framerate such as [Taylor et al 2017], they are more expensive and harder to set up and use compared to marker-based systems. Additionally, such markerless systems cannot yet achieve sub-millimeter accuracy and track the complex hand motions that optical motion capture can, which are required for our real-time dexterous hand interactions for complex tasks and subtle motions.…”

57‐1: Invited Paper: AirTouch System: Hand Skeleton Modelling for Augmented HMI Interactions

“…Our approach lies with using the advantages of both methodologies to compensate for their drawbacks: good initialization of the hand skeleton joint positions by a CNN, which are then better refined by optimization [8,9].…”