Measures of Effective Video Tracking

Abstract

Abstract-To evaluate multi-target video tracking results, one needs to quantify the accuracy of the estimated target-size and the cardinality error as well as measure the frequency of occurrence of ID changes. In this paper we survey existing multi-target tracking performance scores and, after discussing their limitations, we propose three parameter-independent measures for evaluating multi-target video tracking. The measures take into account target-size variations, combine accuracy and cardinality errors, quantify long-term tracking accuracy at different accuracy levels, and evaluate ID changes relative to the duration of the track in which they occur. We conduct an extensive experimental validation of the proposed measures by comparing them with existing ones and by evaluating four state-of-the-art trackers on challenging real-world publicly-available datasets. The software implementing the proposed measures is made available online to facilitate their use by the research community.

A method for performance diagnosis and evaluation of video trackers

“…1b). Here, we consider these three frame-level faults (ID change, false positive, false negative) as they often implicitly or explicitly form a basis for, or contribute to, estimating existing track-level assessment proposals [4][5][6]15,19,22] (see details in Sect. 2).…”

“…Evaluation measures [4][5][6]9,15,19,22] are important techniques of providing a means to draw performance comparisons among different multi-target tracking algorithms [3,17,18,20,21]. These measures are generally aimed to determine end performance of trackers.…”

“…These measures are generally aimed to determine end performance of trackers. End performance provides an overall quantification of goodness or badness of trackers' results in the form of a score at frame level [12,19], or sequence level [4,15], without separately analyzing in an explicit manner the key factors (i.e., false positives, false negatives, ID changes [4]) that contribute to the achievement of a certain performance score. Analysis of these contributory factors may indeed be needed in interpreting performance behavior of tracking algorithms against a variety of datasets.…”

“…Existing measures are broadly made up of composite error counts [4,15], tracking success counts [5,12], tracking failure counts [11,14] and temporal averaging of scores [13,15]; providing tracking quality measurements without giving an explicit insight as why the performance of a tracker is less than perfect. Consider, for example, two cases where a wellknown existing measure, Multiple Object Tracking Accuracy (MOTA) [4], provides a comparable performance for a pair of multi-target trackers on a dataset, and ranks one tracker to be better than the other on another dataset (Fig.…”

Effective evaluation of privacy protection techniques in visible and thermal imagery

“…Moreover, the proposed method does not require target detection and recognition accuracies in the evaluation procedure as in existing methods [30][31][32]34 and is therefore annotation free. Furthermore, while evaluation criteria exist for assessing methods in other computer vision areas, including optical flow estimation, 35 stereo correspondence estimation, 36 and video tracking, 37,38 there is an absence of an established method for the evaluation of different aspects of privacy protection methods. An initiative was made in the form of a challenge for assessing privacy protection techniques under the MediaEval workshops that, however, used an evaluation that mainly relied on subjective judgments.…”