A dataset and evaluation methodology for template-based tracking algorithms

Abstract

Unlike dense stereo, optical flow or multi-view stereo, templatebased tracking lacks benchmark datasets allowing a fair comparison between state-of-the-art algorithms. Until now, in order to evaluate objectively and quantitatively the performance and the robustness of template-based tracking algorithms, mainly synthetically generated image sequences were used. The evaluation is therefore often intrinsically biased.In this paper, we describe the process we carried out to perform the acquisition of real scene image sequences with very precise and accurate ground truth poses using an industrial camera rigidly mounted on the end-effector of a high-precision robotic measurement arm. For the acquisition, we considered most of the critical parameters that influence the tracking results such as: the texture richness and the texture repeatability of the objects to be tracked, the camera motion and speed, and the changes of the object scale in the images and variations of the lighting conditions over time.We designed an evaluation scheme for object detection and interframe tracking algorithms and used the image sequences to apply this scheme to several state-of-the-art algorithms. The image sequences will be made freely available for testing, submitting and evaluating new template-based tracking algorithms, i.e. algorithms that detect or track a planar object in an image sequence given only one image of the object (called the template).

Realtime Registration-Based Tracking via Approximate Nearest Neighbour Search

“…Lieberknecht et al [17] provide a set of benchmark videos designed specifically for comparing 2D tracking algorithms. The videos feature 8 different target objects in videos exemplifying each of the following: angle, range, fast far, fast close, and illumination.…”

Stable Keypoint Recognition using Viewpoint Generative Learning

“…When the distance is below a threshold (2 pixels), the matching is accepted as correct match in this all experiments. In addition we introduce an evaluation measure (Lieberknecht et al, 2009) that performs pose estimation. The precision is a meaningful criterion with which we can compare the different local feature descriptors.…”

Accurate real-time tracking using mutual information

“…The positioning error err is defined as the RMS distance between the correct position of some reference points x * i = w(x i , p * ) and the current position of the points w(x i , p) [11]. The reference points are simply chosen as the 4 corners of the template so that the error becomes:…”

“…To have a quantitative measure of its accuracy and robustness, the tracker has been evaluated on some very demanding reference datasets proposed by Metaio GmbH [11]. Those datasets include a large set of sequences with the typical motions that we are suppose to face in augmented reality applications.…”

“…The tracker is considered in convergence if the error between the estimation and the ground truth is below a given threshold. The error measure is similar to the one defined in equation (29), a detailed definition is available in [11]. The mutual information based tracker proves its robustness and accuracy on most of the sequences.…”