Fully Convolutional Scene Graph Generation

Abstract

Scene Graph Generation (SGG) has achieved significant progress recently. However, most previous works rely heavily on fixed-size entity representations based on bounding box proposals, anchors, or learnable queries. As each representation's cardinality has different trade-offs between performance and computation overhead, extracting highly representative features efficiently and dynamically is both challenging and crucial for SGG. In this work, a novel architecture called RepSGG is proposed to address the aforementioned challenges, formulating a subject as queries, an object as keys, and their relationship as the maximum attention weight between pairwise queries and keys. With more fine-grained and flexible representation power for entities and relationships, RepSGG learns to sample semantically discriminative and representative points for relationship inference. Moreover, the long-tailed distribution also poses a significant challenge for generalization of SGG. A run-time performance-guided logit adjustment (PGLA) strategy is proposed such that the relationship logits are modified via affine transformations based on run-time performance during training. This strategy encourages a more balanced performance between dominant and rare classes. Experimental results show that RepSGG achieves the state-of-the-art or comparable performance on the Visual Genome and Open Images V6 datasets with fast inference speed, demonstrating the efficacy and efficiency of the proposed methods.

RelTR: Relation Transformer for Scene Graph Generation

“…than another one-stage model FCSGG [60]. Our model is also competitive compared with recent two-stage models, and outperforms state-of-the-art visual-based methods.…”

“…Compared to the boom of two-stage approaches, onestage approaches are still in their infancy and have the advantage of being simple, fast and easy to train. To the best of our knowledge, FCSGG [60] is currently the only one-stage scene graph generation framework that encodes objects as box center points and relationships as 2D vector fields. While FCSGG model being lightweight and fast speed, it has a significant performance gap compared to other twostage methods.…”

“…Method 𝐴𝑃50 PredCLS SGCLS SGDET #params(M) FPS R@20 R@50 mR@20 mR@50 R@20 R@50 mR@20 mR@50 R@20 R@50 mR@20 mR@50 Note that the number of FCSGG is directly taken from [60] due to unavailable code.…”

“…We compare scores of R@𝐾 and mR@𝐾, number of parameters and inference speed on SGDET (FPS) with several two-stage models and one-stage model FCSGG [60] in Table 1. Models that not only use visual appearance, but also prior knowledge (e.g.…”

Self-Supervised Road Layout Parsing with Graph Auto-Encoding

“…Scene graphs With the aforementioned goal, scene graphs, were pioneered by [4], where detected objects from input images are modeled as nodes and semantic relations between them are modeled as edges with semantic relational labels. Recently, more approaches [8], [19], [20], [21] are proposed on top of this and deliver state-of-the-art performance and efficiency on public benchmarks like Visual Genome [12]. Naturally, all of them rely on ground truth provided by the dataset for training deep scene graph generation models, and this holds for each new domain-specific task.…”