Composite Relationship Fields with Transformers for Scene Graph Generation

Abstract

Scene graph generation (SGG) methods extract relationships between objects. While most methods focus on improving top-down approaches, which build a scene graph based on detected objects from an off-the-shelf object detector, there is a limited amount of work on bottom-up approaches, which jointly detect objects and their relationships in a single stage.In this work, we present a novel bottom-up SGG approach by representing relationships using Composite Relationship Fields (CoRF). CoRF turns relationship detection into a dense regression and classification task, where each cell of the output feature map identifies surrounding objects and their relationships. Furthermore, we propose a refinement head that leverages Transformers for global scene reasoning, resulting in more meaningful relationship predictions. By combining both contributions, our method outperforms previous bottom-up methods on the Visual Genome dataset by 26% while preserving real-time performance.

Fully Convolutional Scene Graph Generation

“…Instead of pooling features of various shapes, extracting features at multiple pixels is much faster and consumes less memory. Several works [17], [18], [19] explore such point-based entity representation for SGG. Pixel2Graph [17] grounds edges at the midpoints between the bounding box centers of subjects and objects (referred to as subject and object centers for the rest of the paper).…”

“…2) Point-based: single-pixel features extracted from bounding box centers. These methods [17], [18], [19] utilize anchor-free detectors [11], [13], [14] to ground entities and relationships in a regression fashion. 3) Query-based: fixed-size learnable embeddings.…”

“…Regarding the relation representations, most works use compositional contextual features to perform predicate classification. Few researchers [18], [19] have explored different ways to represent relations in a regression fashion. These methods represent entities as keypoints (e.g., entity centers), and relationships as points [17] or vectors [18], [19].…”

“…Few researchers [18], [19] have explored different ways to represent relations in a regression fashion. These methods represent entities as keypoints (e.g., entity centers), and relationships as points [17] or vectors [18], [19]. By regressing and grounding entities and relations geometrically, these methods achieve faster inference speed which is useful for down-stream tasks.…”

“…By regressing and grounding entities and relations geometrically, these methods achieve faster inference speed which is useful for down-stream tasks. However, regression with handcrafted targets [18], [19] is deficient, especially on sparselyannotated datasets [24]. Consequently, the performance of regression is inferior compared to predicate classification.…”