Layout and Context Understanding for Image Synthesis with Scene Graphs

Talavera, Arces; Tan, Daniel Stanley; Azcarraga, Arnulfo P.; Hua, Kai-Lung

doi:10.1109/icip.2019.8803182

Cited by 11 publications

(2 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Scene graphs have been proposed in [1] for the task of image retrieval and attract increasing attention in computer vision and natural language processing communities for different scene understanding tasks such as image captioning [21], [22], [23], VQA [24], [25], [26] and image synthesis [27], [28], [29]. The main purpose of scene graph generation (SGG) is to detect the relationships between objects in the scene.…”

Section: Scene Graph Generationmentioning

confidence: 99%

RelTR: Relation Transformer for Scene Graph Generation

Yang¹,

Yang²,

Rosenhahn³

2022

Preprint

View full text Add to dashboard Cite

Different objects in the same scene are more or less related to each other, but only a limited number of these relationships are noteworthy. Inspired by DETR, which excels in object detection, we view scene graph generation as a set prediction problem and propose an end-to-end scene graph generation model RelTR which has an encoder-decoder architecture. The encoder reasons about the visual feature context while the decoder infers a fixed-size set of triplets subject-predicate-object using different types of attention mechanisms with coupled subject and object queries. We design a set prediction loss performing the matching between the ground truth and predicted triplets for the end-to-end training. In contrast to most existing scene graph generation methods, RelTR is a one-stage method that predicts a set of relationships directly only using visual appearance without combining entities and labeling all possible predicates. Extensive experiments on the Visual Genome and Open Images V6 datasets demonstrate the superior performance and fast inference of our model.

show abstract

Section: Scene Graph Generationmentioning

confidence: 99%

RelTR: Relation Transformer for Scene Graph Generation

Yang¹,

Yang²,

Rosenhahn³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Conditional image generation (Isola et al 2017) has gained popularity in computer vision for its ability of making generative modeling more controllable, as well as its potential of cognitively understanding the visual world. Previous works, for the most part, put their efforts into incorporating the condition with variety of scene descriptions, such as natural language instructions (Tan, Feng, and Ordonez 2019), bounding boxes (Zhao et al 2019;Sun and Wu 2019;Talavera et al 2019), semantic segmentations (Reed et al 2016;Li et al 2019), scene graphs (Johnson et al 2018;Herzig et al 2020), and many more. Although it is often reasonable to combine as many types of conditions as possible, there is also substantial motivation to use as little conditions as possible.…”

Section: Introductionmentioning

confidence: 99%

Scene Graph to Image Synthesis via Knowledge Consensus

Wu,

Wei,

Lin

2023

AAAI

View full text Add to dashboard Cite

In this paper, we study graph-to-image generation conditioned exclusively on scene graphs, in which we seek to disentangle the veiled semantics between knowledge graphs and images. While most existing research resorts to laborious auxiliary information such as object layouts or segmentation masks, it is also of interest to unveil the generality of the model with limited supervision, moreover, avoiding extra cross-modal alignments. To tackle this challenge, we delve into the causality of the adversarial generation process, and reason out a new principle to realize a simultaneous semantic disentanglement with an alignment on target and model distributions. This principle is named knowledge consensus, which explicitly describes a triangle causal dependency among observed images, graph semantics and hidden visual representations. The consensus also determines a new graph-to-image generation framework, carried on several adversarial optimization objectives. Extensive experimental results demonstrate that, even conditioned only on scene graphs, our model surprisingly achieves superior performance on semantics-aware image generation, without losing the competence on manipulating the generation through knowledge graphs.

show abstract