Real-Time 3D Model Tracking in Color and Depth on a Single CPU Core

Abstract

We present a novel method to track 3D models in color and depth data. To this end, we introduce approximations that accelerate the state-of-the-art in region-based tracking by an order of magnitude while retaining similar accuracy. Furthermore, we show how the method can be made more robust in the presence of depth data and consequently formulate a new joint contour and ICP tracking energy. We present better results than the state-of-the-art while being much faster then most other methods and achieving all of the above on a single CPU core.

Deep Model-Based 6D Pose Refinement in RGB

“…Consequently, [38] showed how to improve the color segmentation by using local color histograms over time. Orthogonally, the work [18] approximates the model pose space to avoid GPU computations and enables real-time performance on a single CPU core. All these approaches share the property that they rely on hand-crafted segmentation methods that can fail in the case of sudden appearance changes or occlusion.…”

“…To fulfill the first two properties we propose to align the object contours. Tracking the 6D pose of objects via projective contours has been presented before [18,37,27] but, to the best of our knowledge, has not so far been introduced in a deep learning framework. Contour tracking allows to reduce the difficult problem of 3D geometric alignment to a simpler task of 2D silhouette matching by moving through a distance transform, avoiding explicit correspondence search.…”

“…We therefore propose here a novel formulation to drive the network optimization successfully through the ambiguous 6D solution space. We employ an idea, introduced in [18], that allows us to use an approximate contour for optimization without iterative rendering. When creating a training sample, we use the depth map of the rendering to compute a 3D point cloud in camera space and sample a sparse point set on the contour, denoted as V := {v ∈ R 3 }.…”

“…It is worth mentioning that mixtures of the two sets of methods have been proposed as well [30,6,31,24]. Recently, methods that use only depth [34] or both modalities [21,18,10] have shown that depth can make tracking more robust by providing more clues about occlusion and scale. This work aims to explore how RGB information alone can be sufficient to perform visual tasks such as 3D tracking and 6-Degree-of-Freedom (6DoF) pose refinement by means of a Convolutional Neural Network (CNN).…”

“…As a major contribution we provide a differentiable formulation of a new visual loss that aligns object contours and implicitly optimizes for metric translation and rotation. While our optimization is inspired by region-based approaches, we can track objects of any texture or shape since we do not need to model global [27,37,18] or local appearance [11,38]. Instead, we show that we can do away with these hand-crafted approaches by letting the network learn the object appearance implicitly.…”