PPDL: Predicate Probability Distribution based Loss for Unbiased Scene Graph Generation

Li, Wei; Zhang, Haiwei; Bai, Qijie; Zhao, Guangyao; Jiang, Ning; Yuan, Xiaojie

doi:10.1109/cvpr52688.2022.01884

Cited by 27 publications

(11 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…[21,33,39,42,43]. Recently, a number of papers have identified the long-tailed distribution in image scene graphs and focused on generating unbiased scene graphs [8,9,[15][16][17]41]. We seek to bring the same problem to light in the video domain.…”

Section: Related Workmentioning

confidence: 99%

Multi-Label Meta Weighting for Long-Tailed Dynamic Scene Graph Generation

Chen

Mettes³

et al. 2023

Proceedings of the 2023 ACM International Conference on Multimedia Retrieval

View full text Add to dashboard Cite

This paper investigates the problem of scene graph generation in videos with the aim of capturing semantic relations between subjects and objects in the form of ⟨subject, predicate, object⟩ triplets. Recognizing the predicate between subject and object pairs is imbalanced and multi-label in nature, ranging from ubiquitous interactions such as spatial relationships (e.g. in front of ) to rare interactions such as twisting. In widely-used benchmarks such as Action Genome and VidOR, the imbalance ratio between the most and least frequent predicates reaches 3,218 and 3,408, respectively, surpassing even benchmarks specifically designed for long-tailed recognition. Due to the long-tailed distributions and label co-occurrences, recent state-of-the-art methods predominantly focus on the most frequently occurring predicate classes, ignoring those in the long tail. In this paper, we analyze the limitations of current approaches for scene graph generation in videos and identify a one-to-one correspondence between predicate frequency and recall performance. To make the step towards unbiased scene graph generation in videos, we introduce a multi-label meta-learning framework to deal with the biased predicate distribution. Our meta-learning framework learns a meta-weight network for each training sample over all possible label losses. We evaluate our approach on the Action Genome and VidOR benchmarks by building upon two current state-of-theart methods for each benchmark. The experiments demonstrate that the multi-label meta-weight network improves the performance for predicates in the long tail without compromising performance for head classes, resulting in better overall performance and favorable generalizability. Code: https://github.com/shanshuo/ML-MWN. CCS CONCEPTS• Computing methodologies → Scene understanding; Activity recognition and understanding.

show abstract

Section: Related Workmentioning

confidence: 99%

Multi-Label Meta Weighting for Long-Tailed Dynamic Scene Graph Generation

Chen

Mettes³

et al. 2023

Proceedings of the 2023 ACM International Conference on Multimedia Retrieval

View full text Add to dashboard Cite

show abstract

“…However, these methods suffered from head-biased predictions due to the inherent imbalance of predicates. In recent years, several unbiased approaches [20]- [24], [26]- [28], [35]- [41] have been proposed to address this problem. For example, TDE [20] uses causal analysis to eliminate prediction bias, Cogtree [22] employs cognitive tree loss for unbiased prediction, while BGNN [24] balances head and tail predicates through bi-level re-sampling.…”

Section: Related Work a Unbiased Scene Graph Generationmentioning

confidence: 99%

“…To evaluate our proposed method on VG150, our approach combines four classical SGG models to represent predicate features, namely Motifs [29], VCTree [30], Transformer [49] and PENet [31], as shown in Table II. The results of the state-of-the-art methods that are being compared are divided into various debiased methods [20], [22], [26]- [28], [35], [57], [58], [60], [61] on classical models and specific SGG models [21], [23], [24], [27], [32], [56], [59].…”

Section: Comparison With State Of the Arts A) Vg150mentioning

confidence: 99%

Learning to Generate an Unbiased Scene Graph by Using Attribute-Guided Predicate Features

Wang

Yuan

Chen

2023

AAAI

View full text Add to dashboard Cite

Scene Graph Generation (SGG) aims to capture the semantic information in an image and build a structured representation, which facilitates downstream tasks. The current challenge in SGG is to tackle the biased predictions caused by the long-tailed distribution of predicates. Since multiple predicates in SGG are coupled in an image, existing data re-balancing methods cannot completely balance the head and tail predicates. In this work, a decoupled learning framework is proposed for unbiased scene graph generation by using attribute-guided predicate features to construct a balanced training set. Specifically, the predicate recognition is decoupled into Predicate Feature Representation Learning (PFRL) and predicate classifier training with a class-balanced predicate feature set, which is constructed by our proposed Attribute-guided Predicate Feature Generation (A-PFG) model. In the A-PFG model, we first define the class labels of and corresponding visual feature as attributes to describe a predicate. Then the predicate feature and the attribute embedding are mapped into a shared hidden space by a dual Variational Auto-encoder (VAE), and finally the synthetic predicate features are forced to learn the contextual information in the attributes via cross reconstruction and distribution alignment. To demonstrate the effectiveness of our proposed method, our decoupled learning framework and A-PFG model are applied to various SGG models. The empirical results show that our method is substantially improved on all benchmarks and achieves new state-of-the-art performance for unbiased scene graph generation. Our code is available at https://github.com/wanglei0618/A-PFG.

show abstract

“…Long-tailed data distribution has been a key challenge in visual recognition [41], and it has been addressed in the recent literature on SGG [42]. In order to tackle this problem, various approaches have been proposed, such as data resampling [43], [44], [45], [46], de-biasing [16], [47], [48], [49], [50], and loss modification [51], [52], [53], [54], [55], [56], [57]. De-biasing methods require pre-trained biased models for initialization and then finetune the model.…”

Section: Long-tailed Distributionsmentioning

confidence: 99%

Fully Convolutional Scene Graph Generation

Liu

Yan

Mortazavi

et al. 2021

2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

View full text Add to dashboard Cite

Scene Graph Generation (SGG) has achieved significant progress recently. However, most previous works rely heavily on fixed-size entity representations based on bounding box proposals, anchors, or learnable queries. As each representation's cardinality has different trade-offs between performance and computation overhead, extracting highly representative features efficiently and dynamically is both challenging and crucial for SGG. In this work, a novel architecture called RepSGG is proposed to address the aforementioned challenges, formulating a subject as queries, an object as keys, and their relationship as the maximum attention weight between pairwise queries and keys. With more fine-grained and flexible representation power for entities and relationships, RepSGG learns to sample semantically discriminative and representative points for relationship inference. Moreover, the long-tailed distribution also poses a significant challenge for generalization of SGG. A run-time performance-guided logit adjustment (PGLA) strategy is proposed such that the relationship logits are modified via affine transformations based on run-time performance during training. This strategy encourages a more balanced performance between dominant and rare classes. Experimental results show that RepSGG achieves the state-of-the-art or comparable performance on the Visual Genome and Open Images V6 datasets with fast inference speed, demonstrating the efficacy and efficiency of the proposed methods.

show abstract

PPDL: Predicate Probability Distribution based Loss for Unbiased Scene Graph Generation

Cited by 27 publications

References 24 publications

Multi-Label Meta Weighting for Long-Tailed Dynamic Scene Graph Generation

Multi-Label Meta Weighting for Long-Tailed Dynamic Scene Graph Generation

Learning to Generate an Unbiased Scene Graph by Using Attribute-Guided Predicate Features

Fully Convolutional Scene Graph Generation

Contact Info

Product

Resources

About