Scene graph captioner: Image captioning based on structural visual representation

Xu, Ning; Liu, An-An; Liu, Jing; Nie, Weizhi; Su, Yuting

doi:10.1016/j.jvcir.2018.12.027

Cited by 80 publications

(33 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…representations have been used for multiple computer vision applications, including image retrieval [52], image captioning [53], and image generation [54]. In our work, we employ the rule-based scene graph parser from [55] to extract object candidates and their corresponding attributes from captions, and make use of this information to train a localization network for WSSS.…”

Section: F Scene Graphsmentioning

confidence: 99%

Extracting Structured Supervision From Captions for Weakly Supervised Semantic Segmentation

Vilar

Perez

2021

IEEE Access

View full text Add to dashboard Cite

Weakly supervised semantic segmentation (WSSS) methods have received significant attention in recent years, since they can dramatically reduce the annotation costs of fully supervised alternatives. While most previous studies focused on leveraging classification labels, we explore instead the use of image captions, which can be obtained easily from the web and contain richer visual information. Existing methods for this task assigned text snippets to relevant semantic labels by simply matching class names, and then employed a model trained to localize arbitrary text in images to generate pseudoground truth segmentation masks. Instead, we propose a dedicated caption processing module to extract structured supervision from captions, consisting of improved relevant object labels, their visual attributes, and additional background categories, all of which are useful for improving segmentation quality. This module uses syntactic structures learned from text data, and semantic relations retrieved from a knowledge database, without requiring additional annotations on the specific image domain, and consequently can be extended immediately to new object categories. We then present a novel localization network, which is trained to localize only these structured labels. This strategy simplifies model design, while focusing training signals on relevant visual information. Finally, we describe a method for leveraging all types of localization maps to obtain high-quality segmentation masks, which are used to train a supervised model. On the challenging MS-COCO dataset, our method moves the state-of-the-art forward significantly for WSSS with image-level supervision by a margin of 7.6% absolute (26.7% relative) mean Intersection-over-Union, achieving 54.5% precision and 50.9% recall.

show abstract

Section: F Scene Graphsmentioning

confidence: 99%

Extracting Structured Supervision From Captions for Weakly Supervised Semantic Segmentation

Vilar

Perez

2021

IEEE Access

View full text Add to dashboard Cite

show abstract

“…These works could be sorted based on the attention network structure and the attention weight calculation method. (1) Single layer vs. multi-layer (see, for example, studies [3,3,4,[19][20][21]) employs a single layer implement of an attention mechanism by taking the hidden state as the query vector to extract visual features at each step, while the studies [8,16,17,22,23] chose a multi-layer attention implementation in their decoder. (2) Involving extra clues in attention weights or not-for example, studies [11,24,25] obtained their attention weights only from previous attention calculations, while the authors of [18] combined geometry clues with previously calculated weights into final attention weights.…”

Section: Related Workmentioning

confidence: 99%

Learn and Tell: Learning Priors for Image Caption Generation

2020

View full text Add to dashboard Cite

In this work, we propose a novel priors-based attention neural network (PANN) for image captioning, which aims at incorporating two kinds of priors, i.e., the probabilities being mentioned for local region proposals (PBM priors) and part-of-speech clues for caption words (POS priors), into a visual information extraction process at each word prediction. This work was inspired by the intuitions that region proposals have different inherent probabilities for image captioning, and that the POS clues bridge the word class (part-of-speech tag) with the categories of visual features. We propose new methods to extract these two priors, in which the PBM priors are obtained by computing the similarities between the caption feature vector and local feature vectors, while the POS priors are predicated at each step of word generation by taking the hidden state of the decoder as input. After that, these two kinds of priors are further incorporated into the PANN module of the decoder to help the decoder extract more accurate visual information for the current word generation. In our experiments, we qualitatively analyzed the proposed approach and quantitatively evaluated several captioning schemes with our PANN on the MS-COCO dataset. Experimental results demonstrate that our proposed method could achieve better performance as well as the effectiveness of the proposed network for image captioning.

show abstract

“…Recently, increasing researchers are engaged into generating the scene graph [40], where the node denotes the object and the edge represents the relationship between pairwise nodes. Due to the comprehensive and coherent visually-grounded knowledge, the scene graph can contribute to a variety of AI tasks, such as image retrieval [13], image captioning [41,45], image generation [12], and visual Q&A [21,25,36]. The scene graph is formed of directed edges, which connect two objects as the ⟨subject-predicate-object ⟩ triplet.…”

Section: Introductionmentioning

confidence: 99%

Part-Aware Interactive Learning for Scene Graph Generation

Tian

Liu

et al. 2020

Proceedings of the 28th ACM International Conference on Multimedia

Self Cite

View full text Add to dashboard Cite

Generating scene graph to describe the whereabouts and interactions of objects in an image has attracted increasing attention of researchers. Most existing methods explore object-level visual context or bodypart-object cooperation with the message passing structure, which can not meet the part-aware interaction nature of scene graph. Normally, a subject interacts with an object through crucial parts in each other. Besides, the correlation among parts within an identical object can also help predicting objects and their relationships. Hence, both of subject and object parts and their intra-and inter-object correlations should be fully considered for scene graph generation. In this paper, we propose a part-aware interactive learning method, which are divided into the intra-object and inter-object scenarios. First, we detect objects from an image and further decompose each one into a set of parts. Second, the part-aware graph attention module is proposed to refine part features via the intra-object message passing, and the refined features are incorporated for object inference. Third, the visual mutual attention module is designed to discover part-aware correlated visual cues precisely for predicate inference. It can highlight the subjectrelated object parts and the object-related subject parts during inter-object interactive learning. We demonstrate the superiority of our method against the state of the arts on Visual Genome. Ablation studies and visualization further validate its effectiveness. CCS CONCEPTS • Computing methodologies → Scene understanding.

show abstract

Scene graph captioner: Image captioning based on structural visual representation

Cited by 80 publications

References 17 publications

Extracting Structured Supervision From Captions for Weakly Supervised Semantic Segmentation

Extracting Structured Supervision From Captions for Weakly Supervised Semantic Segmentation

Learn and Tell: Learning Priors for Image Caption Generation

Part-Aware Interactive Learning for Scene Graph Generation

Contact Info

Product

Resources

About