Recovering the Unbiased Scene Graphs from the Biased Ones

Abstract

Given input images, scene graph generation (SGG) aims to produce comprehensive, graphical representations describing visual relationships among salient objects. Recently, more efforts have been paid to the long tail problem in SGG; however, the imbalance in the fraction of missing labels of different classes, or reporting bias, exacerbating the long tail is rarely considered and cannot be solved by the existing debiasing methods. In this paper we show that, due to the missing labels, SGG can be viewed as a "Learning from Positive and Unlabeled data" (PU learning) problem, where the reporting bias can be removed by recovering the unbiased probabilities from the biased ones by utilizing label frequencies, i.e., the per-class fraction of labeled, positive examples in all the positive examples. To obtain accurate label frequency estimates, we propose Dynamic Label Frequency Estimation (DLFE) to take advantage of training-time data augmentation and average over multiple training iterations to introduce more valid examples. Extensive experiments show that DLFE is more effective in estimating label frequencies than a naive variant of the traditional estimate, and DLFE significantly alleviates the long tail and achieves state-of-the-art debiasing performance on the VG dataset. We also show qualitatively that SGG models with DLFE produce prominently more balanced and unbiased scene graphs. The source code will be publicly available 1 .

SGTR: End-to-end Scene Graph Generation with Transformer

“…On dense relationship proposals, some works propose a contextual modeling structure [11, 19-21, 23, 27, 33, 38, 39, 41, 42, 45, 50-55]. Recent studies have focused on developing logit adjustment and training strategies to address the SGG task's long-tail recognition [1,4,5,7,14,15,19,29,32,38,43,44,48]. The twostage design is capable of dealing with the complicated scenarios encountered in SGG.…”

“…The two components of the total cost correspond to the cost of the predicate, subject, and object entity, respectively. 4 The matching index I tri between the triplet prediction and the ground truth is produced by: I tri = argmin T ,T gt C, which is used for the following loss calculation of the predicate node generator. The two terms of L pre , that is, L pre i , L pre p are used to supervise two types of sub-decoder in predicate node generator.…”

“…The two terms of L pre , that is, L pre i , L pre p are used to supervise two types of sub-decoder in predicate node generator. For the entity indicator sub-decoder, we have We further remove the invalid self-connection edges during 4 We utilize the location and classification predictions to calculate the cost for each component. Detailed formulations are presented in the supplementary.…”

“…Most previous works attempt to tackle this problem using two distinct design patterns: bottom-up two-stage [1,4,5,7,15,19,44,48] and point-based one-stage design [6,24]. The former design typically first detects N entity proposals, followed by predicting the predicate categories of those entity combinations.…”

“…Our model has a modular architecture consisting of four main submodules: (1) a backbone network to generate feature representation of the scene (Sec 4…”