Performance evaluation of real-time video content analysis systems in the CANDELA project

Abstract

The CANDELA project aims at realizing a system for real-time image processing in traffic and surveillance applications. The system performs segmentation, labels the extracted blobs and tracks their movements in the scene. Performance evaluation of such a system is a major challenge since no standard methods exist and the criteria for evaluation are highly subjective. This paper proposes a performance evaluation approach for video content analysis (VCA) systems and identifies the involved research areas. For these areas we give an overview of the state-of-the-art in performance evaluation and introduce a classification into different semantic levels. The proposed evaluation approach compares the results of the VCA algorithm with a ground-truth (GT) counterpart, which contains the desired results. Both the VCA results and the ground truth comprise description files that are formatted in MPEG-7. The evaluation is required to provide an objective performance measure and a mean to choose between competitive methods. In addition, it enables algorithm developers to measure the progress of their work at the different levels in the design process. From these requirements and the state-of-the-art overview we conclude that standardization is highly desirable for which many research topics still need to be addressed.

Nonintrusive viewpoint tracking for 3D for perception in smart video conference

“…Please refer to [19] for a review of performance evaluation in vision. For faces / eyes detection, Popovici et al [20] propose a scoring function to both validate a detection and quantify the goodness of eyes position.…”

On the performance evaluation of tracking systems using multiple pan-tilt-zoom cameras

“…In our case, the "working region" is considered as the points of the stadium ground plane that are visible in the current screen image. Thus we propose the metric H error to characterize the error of the homography determination (see (2) ).…”

“…The first requirement for an evaluation of a VCA algorithm is to have video data. Indeed, to enable proper benchmarking with other algorithms, it makes sense to evaluate algorithms with standard video data [2]. For the TRICTRAC project, public sequences were available like from the IST Inmove 2 project (see figure 2) but were done with static cameras.…”

Flexible surveillance system architecture for prototyping video content analysis algorithms

“…To monitor the development of VCA algorithms, or to compare multiple VCA algorithms, benchmarking of the system is important 10 . To benchmark each module in such a system, local performance evaluation is proposed.…”