The Overlooked Elephant of Object Detection: Open Set

Towards Open-Set Scene Graph Generation With Unknown Objects

“…We note that the number of unknown objects classified as known ("1-wrong") coincides with absolute open-set error proposed for open-set object detection [41], which was also used in the recent study proposing a stateof-the-art open-set object detection method [44]. Meanwhile, we do not use precision-like metrics such as another open-set detection metric called wilderness ratio [43] since they are not suitable to the sparsely annotated VG dataset as described above.…”

“…Thus, we present novel experimental results for relationship-aware open-set object detection and relationship detection, both of which have not been evaluated by the previous studies. In addition, unlike a recent evaluation study [43] on open-set detection, we compare several baseline methods, including unknown-aware extensions of existing methods. Although more sophisticated unknown detection techniques have been proposed for openset recognition [39], [40] and object detection [44], we leave integration of such advanced techniques with state-of-the-art SGG methods as a future research topic.…”

“…Another important difference from object detection is that SGG needs a specialized dataset with ground-truth relationship labels for training and testing. Thus, we cannot reuse the open-set detection datasets with unknown objects used in the previous studies, nor follow their dataset construction scheme [43], [44] 1. This table highlights the novelty of this work.…”

“…2, the number of testing images rapidly grows as we randomly add classes to the unknown set, leaving almost no training images. Note that previous studies on open-set object detection did not face this issue, since they could combine multiple datasets with mutually exclusive class definitions to ensure that images from one dataset do not contain the classes from the others [43], [44], while we do not have other large-scale datasets like VG to be combined.…”

“…In addition to these metrics as a natural extension of previous closed-set SGG, we also propose recall-like metrics designed for detailed analysis of object and relationships detection in open-set SGG with unknown objects, inspired by previous studies in open-set object detection [41], [43], [44]. As in the case of the recall-based metrics, we count ground-truth objects or relationships in each image.…”