Humans Disagree With the IoU for Measuring Object Detector Localization Error

Abstract

The localization quality of automatic object detectors is typically evaluated by the Intersection over Union (IoU) score. In this work, we show that humans have a different view on localization quality. To evaluate this, we conduct a survey with more than 70 participants. Results show that for localization errors with the exact same IoU score, humans might not consider that these errors are equal, and express a preference. Our work is the first to evaluate IoU with humans and makes it clear that relying on IoU scores alone to evaluate localization errors might not be sufficient.

Improving Few-Shot Object Detection through a Performance Analysis on Aerial and Natural Images

“…It should be preferred over IoU to assess the performance of models on all visual tasks that commonly employ IoU within their evaluation process. It supports recent findings that show misalignment between IoU and human preference [32].…”

“…Finally, even though SIoU aligns better than IoU with human perception, it does not match completely with it. IoU and SIoU do not account for object content whereas humans heavily do, as highlighted by [32].…”

“…However, IoU is involved in other parts of the framework including example selection, Non-Maximal Suppression, and evaluation. A recent user study [32] indicates that IoU does not completely align with human perception. Humans have strong positional and size preferences based on conceptual information contained in the boxes.…”

“…Only a handful of works question the adequation of IoU for object detection. Among those, [28] proposed a generalization of IoU when boxes do not overlap, [40] introduced a novel loss function to target small objects and [32] showed that human perception and IoU are not fully aligned. This lack of interest in new criterion design is explained by the great detection performance in the regular setting (i.e.…”