HOSE-Net: Higher Order Structure Embedded Network for Scene Graph Generation

Meng, Wei; Yuan, Can; Yue, Xiaoyu; Zhong, Kuo

doi:10.1145/3394171.3413575

Cited by 16 publications

(11 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This approach controls the weight on different sample types, diminishing the sideeffect of the negative samples on prediction. Wei et al [186] proposed HOSENet to concentrate on the semantic overlap between the objects, by building two object sets (L s ,L o ) and calculating the similarity between subject and object within a triplet. Wang et al [195] believed that larger targets represent major relationship positions in a scene, thus they built a tree-like structure and a Relation Ranking Module (RRM) to pay more attention to the relationships and objects within the salient region.…”

Section: Other Sgg Methodsmentioning

confidence: 99%

Scene Graph Generation: A Comprehensive Survey

Zhu¹,

Zhang²,

Jiang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Deep learning techniques have led to remarkable breakthroughs in the field of generic object detection and have spawned a lot of scene-understanding tasks in recent years. Scene graph has been the focus of research because of its powerful semantic representation and applications to scene understanding. Scene Graph Generation (SGG) refers to the task of automatically mapping an image into a semantic structural scene graph, which requires the correct labeling of detected objects and their relationships. Although this is a challenging task, the community has proposed a lot of SGG approaches and achieved good results. In this paper, we provide a comprehensive survey of recent achievements in this field brought about by deep learning techniques. We review 138 representative works that cover different input modalities, and systematically summarize existing methods of image-based SGG from the perspective of feature extraction and fusion. We attempt to connect and systematize the existing visual relationship detection methods, to summarize, and interpret the mechanisms and the strategies of SGG in a comprehensive way. Finally, we finish this survey with deep discussions about current existing problems and future research directions. This survey will help readers to develop a better understanding of the current research status and ideas.

show abstract

Section: Other Sgg Methodsmentioning

confidence: 99%

Scene Graph Generation: A Comprehensive Survey

Zhu¹,

Zhang²,

Jiang³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Scene graph generation localizes not only objects but also recognizes their relationships, which is a visual task with higher semantic abstraction. As a structural representation of images, the objects and their semantic relationships are represented as nodes and edges in the scene graph [5,[7][8][9]. There are a series of triples, <subject-predicate-object>, in the scene graph, "predicate" represents a specific semantic relationship, and "subject" and "object" are the two instances involved, as shown in Figure 1c.…”

Section: Introduction 1background Of Scene Graph Generationmentioning

confidence: 99%

“…There are a series of triples, <subject-predicate-object>, in the scene graph, "predicate" represents a specific semantic relationship, and "subject" and "object" are the two instances involved, as shown in Figure 1c. As a bridge between low-level recognition and high-level understanding of images, the scene graph supports many downstream vision tasks [8][9][10][11][12], such as image captioning [13,14] and visual question answering [15,16].…”

Section: Introduction 1background Of Scene Graph Generationmentioning

confidence: 99%

RSSGG_CS: Remote Sensing Image Scene Graph Generation by Fusing Contextual Information and Statistical Knowledge

et al. 2022

View full text Add to dashboard Cite

To semantically understand remote sensing images, it is not only necessary to detect the objects in them but also to recognize the semantic relationships between the instances. Scene graph generation aims to represent the image as a semantic structural graph, where objects and relationships between them are described as nodes and edges, respectively. Some existing methods rely only on visual features to sequentially predict the relationships between objects, ignoring contextual information and making it difficult to generate high-quality scene graphs, especially for remote sensing images. Therefore, we propose a novel model for remote sensing image scene graph generation by fusing contextual information and statistical knowledge, namely RSSGG_CS. To integrate contextual information and calculate attention among all objects, the RSSGG_CS model adopts a filter module (FiM) that is based on adjusted transformer architecture. Moreover, to reduce the blindness of the model when searching semantic space, statistical knowledge of relational predicates between objects from the training dataset and the cleaned Wikipedia text is used as supervision when training the model. Experiments show that fusing contextual information and statistical knowledge allows the model to generate more complete scene graphs of remote sensing images and facilitates the semantic understanding of remote sensing images.

show abstract

“…Most of the existing efforts in long-tailed SGG [6,14,37,41,43,45] deal with the skewed class distribution directly. However, 1 https://github.com/coldmanck/recovering-unbiased-scene-graphs Traditionally SGG models are not trained in the PU setting and thus output biased probabilities in favor of conspicuous classes (e.g., on).…”

Section: Introductionmentioning

confidence: 99%

“…To produce meaningful scene graphs, the inconspicuous but informative predicates need to be properly predicted. To the best of our knowledge, none of the existing SGG debiasing methods [6,14,37,41,43,45] effectively solve this reporting bias problem.…”

Section: Introductionmentioning

confidence: 99%

Recovering the Unbiased Scene Graphs from the Biased Ones

Chiou

Ding²,

Yan

et al. 2021

Preprint

View full text Add to dashboard Cite

Given input images, scene graph generation (SGG) aims to produce comprehensive, graphical representations describing visual relationships among salient objects. Recently, more efforts have been paid to the long tail problem in SGG; however, the imbalance in the fraction of missing labels of different classes, or reporting bias, exacerbating the long tail is rarely considered and cannot be solved by the existing debiasing methods. In this paper we show that, due to the missing labels, SGG can be viewed as a "Learning from Positive and Unlabeled data" (PU learning) problem, where the reporting bias can be removed by recovering the unbiased probabilities from the biased ones by utilizing label frequencies, i.e., the per-class fraction of labeled, positive examples in all the positive examples. To obtain accurate label frequency estimates, we propose Dynamic Label Frequency Estimation (DLFE) to take advantage of training-time data augmentation and average over multiple training iterations to introduce more valid examples. Extensive experiments show that DLFE is more effective in estimating label frequencies than a naive variant of the traditional estimate, and DLFE significantly alleviates the long tail and achieves state-of-the-art debiasing performance on the VG dataset. We also show qualitatively that SGG models with DLFE produce prominently more balanced and unbiased scene graphs. The source code will be publicly available 1 .

show abstract

HOSE-Net: Higher Order Structure Embedded Network for Scene Graph Generation

Cited by 16 publications

References 31 publications

Scene Graph Generation: A Comprehensive Survey

Scene Graph Generation: A Comprehensive Survey

RSSGG_CS: Remote Sensing Image Scene Graph Generation by Fusing Contextual Information and Statistical Knowledge

Recovering the Unbiased Scene Graphs from the Biased Ones

Contact Info

Product

Resources

About