A Comparative Analysis of Object Detection Metrics with a Companion Open-Source Toolkit

Abstract

Recent outstanding results of supervised object detection in competitions and challenges are often associated with specific metrics and datasets. The evaluation of such methods applied in different contexts have increased the demand for annotated datasets. Annotation tools represent the location and size of objects in distinct formats, leading to a lack of consensus on the representation. Such a scenario often complicates the comparison of object detection methods. This work alleviates this problem along the following lines: (i) It provides an overview of the most relevant evaluation methods used in object detection competitions, highlighting their peculiarities, differences, and advantages; (ii) it examines the most used annotation formats, showing how different implementations may influence the assessment results; and (iii) it provides a novel open-source toolkit supporting different annotation formats and 15 performance metrics, making it easy for researchers to evaluate the performance of their detection algorithms in most known datasets. In addition, this work proposes a new metric, also included in the toolkit, for evaluating object detection in videos that is based on the spatio-temporal overlap between the ground-truth and detected bounding boxes.

Using a two-stage convolutional neural network to rapidly identify tiny herbivorous beetles in the field

“…This led to the overdetection of background objects and insects. In comparison, the threshold of confidence score for an object detection task by the YOLO series was often set to a value that ranged from 0.3 to 0.5 (Padilla et al, 2021; Ovchinnikova et al, 2021; Redmon et al, 2016; Redmon and Farhadi, 2018).…”

Cost-Performance Evaluation of a Recognition Service of Livestock Activity Using Aerial Images

“…The mean average precision, which is defined in (4), is the result of averaging the AP for each class. AP and mAP has several forms of calculation [31]:…”

An Approximate GEMM Unit for Energy-Efficient Object Detection

“…The performance of an object detection model can be assessed in terms of precision and recall [ 89 ], where , , and are the numbers of true positive, false positive, and false negative detections. Intuitively, precision P measures the accuracy of assigning the correct class label, while recall R measures the accuracy of finding ground truth objects.…”

“…One can roughly assess the performance of an object detector by computing the area under the precision–recall curve. To estimate the area under the curve, the average precision uses N -point interpolation [ 89 ], …”

“…In the further analysis, we used the metric at the threshold and the metric averaged over ten equidistant thresholds . Using mAP-based metrics in combination with cross-validation is a standard approach for performance evaluation and model comparison in object detection benchmarks [ 2 , 89 , 90 , 91 ].…”