Towards Overcoming False Positives in Visual Relationship Detection

Abstract

In this paper, we investigate the cause of the high false positive rate in Visual Relationship Detection (VRD). We observe that during training, the relationship proposal distribution is highly imbalanced: most of the negative relationship proposals are easy to identify, e.g., the inaccurate object detection, which leads to the under-fitting of lowfrequency difficult proposals. This paper presents Spatially-Aware Balanced negative pRoposal sAmpling (SABRA), a robust VRD framework that alleviates the influence of false positives. To effectively optimize the model under imbalanced distribution, SABRA adopts Balanced Negative Proposal Sampling (BNPS) strategy for mini-batch sampling. BNPS divides proposals into 5 well defined sub-classes and generates a balanced training distribution according to the inverse frequency. BNPS gives an easier optimization landscape and significantly reduces the number of false positives. To further resolve the low-frequency challenging false positive proposals with high spatial ambiguity, we improve the spatial modeling ability of SABRA on two aspects: a simple and efficient multi-head heterogeneous graph attention network (MH-GAT) that models the global spatial interactions of objects, and a spatial mask decoder that learns the local spatial configuration. SABRA outperforms SOTA methods by a large margin on two human-object interaction (HOI) datasets and one general VRD dataset.

A Symmetric Fusion Learning Model for Detecting Visual Relations and Scene Parsing

“…VRD: Table 2 presents comparisons on VRD with eight state-of-the-art methods: VRD [3], KL distillation [23], Zoom-net [25], CAI + SCA-M [25], RelDN [37], AVR [42], GPS-Net [43], and SABRA [44]. For a fair comparison of VRD, we adopt the VGG-16 backbone pretrained on ImageNet used for these baselines to train our model.…”

“…Relation detection Phrase detection R@ 50 R@ 100 R@ 50 R@ 100 1 Baseline (20% noisy labels) 18.00 22. 44 > , making it more precise to parse the scene graph. It also can be observed that our model processed more complete detection of visual relationships; some triplets that are not in ground-truth are also precisely detected.…”

Scene Graph Generation: A Comprehensive Survey

“…Beyond that, there are several other noteworthy DL architectures for scene graph generation and visual relationship detection. Jin et al [197] proposed SABRA to classify all the negative triplet samples into five general types and use the Balanced Negative Proposal Sampling to get training samples with a balanced distribution. This approach controls the weight on different sample types, diminishing the sideeffect of the negative samples on prediction.…”

“…Recently, two novel techniques i.e., SABRA [197] and HET [195] have achieved SOTA performance for PhrDet and RelDet on VRD, respectively. SABRA enhanced the robustness of the training process of the proposed model by subdividing negative samples, while HET followed the intuitive perspective i.e., the more salient the object, the more important it would be for the scene graph.…”