Deeply Supervised Multimodal Attentional Translation Embeddings for Visual Relationship Detection

Gkanatsios, Nikolaos; Pitsikalis, Vassilis; Koutras, Petros; Zlatintsi, Athanasia; Maragos, Petros

doi:10.1109/icip.2019.8803106

Cited by 12 publications

(5 citation statements)

References 23 publications

(94 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The Multimodal Attentional Translation Embeddings (MATransE) model built upon VTransE [71] learns a projection of <S, P, O> into a score space where S + P ≈ O, by guiding the features' projection with attention to satisfy:…”

Section: Visual Translation Embeddingmentioning

confidence: 99%

Scene Graph Generation: A Comprehensive Survey

Zhu¹,

Zhang²,

Jiang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Deep learning techniques have led to remarkable breakthroughs in the field of generic object detection and have spawned a lot of scene-understanding tasks in recent years. Scene graph has been the focus of research because of its powerful semantic representation and applications to scene understanding. Scene Graph Generation (SGG) refers to the task of automatically mapping an image into a semantic structural scene graph, which requires the correct labeling of detected objects and their relationships. Although this is a challenging task, the community has proposed a lot of SGG approaches and achieved good results. In this paper, we provide a comprehensive survey of recent achievements in this field brought about by deep learning techniques. We review 138 representative works that cover different input modalities, and systematically summarize existing methods of image-based SGG from the perspective of feature extraction and fusion. We attempt to connect and systematize the existing visual relationship detection methods, to summarize, and interpret the mechanisms and the strategies of SGG in a comprehensive way. Finally, we finish this survey with deep discussions about current existing problems and future research directions. This survey will help readers to develop a better understanding of the current research status and ideas.

show abstract

Section: Visual Translation Embeddingmentioning

confidence: 99%

Scene Graph Generation: A Comprehensive Survey

Zhu¹,

Zhang²,

Jiang³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Long-tailed data distribution has been a key challenge in visual recognition [41], and it has been addressed in the recent literature on SGG [42]. In order to tackle this problem, various approaches have been proposed, such as data resampling [43], [44], [45], [46], de-biasing [16], [47], [48], [49], [50], and loss modification [51], [52], [53], [54], [55], [56], [57]. De-biasing methods require pre-trained biased models for initialization and then finetune the model.…”

Section: Long-tailed Distributionsmentioning

confidence: 99%

Fully Convolutional Scene Graph Generation

Liu

Yan

Mortazavi

et al. 2021

2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

View full text Add to dashboard Cite

Scene Graph Generation (SGG) has achieved significant progress recently. However, most previous works rely heavily on fixed-size entity representations based on bounding box proposals, anchors, or learnable queries. As each representation's cardinality has different trade-offs between performance and computation overhead, extracting highly representative features efficiently and dynamically is both challenging and crucial for SGG. In this work, a novel architecture called RepSGG is proposed to address the aforementioned challenges, formulating a subject as queries, an object as keys, and their relationship as the maximum attention weight between pairwise queries and keys. With more fine-grained and flexible representation power for entities and relationships, RepSGG learns to sample semantically discriminative and representative points for relationship inference. Moreover, the long-tailed distribution also poses a significant challenge for generalization of SGG. A run-time performance-guided logit adjustment (PGLA) strategy is proposed such that the relationship logits are modified via affine transformations based on run-time performance during training. This strategy encourages a more balanced performance between dominant and rare classes. Experimental results show that RepSGG achieves the state-of-the-art or comparable performance on the Visual Genome and Open Images V6 datasets with fast inference speed, demonstrating the efficacy and efficiency of the proposed methods.

show abstract

“…However, simple positional embedding implicitly captures spatial information with position coordinates as inputs to networks, which is unable to capture the explicit spatial configuration in the feature space. [13] introduces binary masks which explicitly specify the subject and object positions, implicitly specifying the spatial configuration by concatenating with union features. With the recent advances in graph neural networks, the relevant positional information can be captured by message passing between instances [59].…”

Section: Spatial Informationmentioning

confidence: 99%

“…Different from the standard positional embedding [57] which implicitly utilizes the spatial information with the absolute positions as an input, SMD explicitly learns a structured embedding. Compared to [13] which concatenates binary mask as position feature, SMD explicitly imposes the spatial structure in the embedding vector and gives better spatially-aware embeddings. We empirically show SMD outperforms these variants in our experiments.…”

Section: Spatial Mask Decodermentioning

confidence: 99%

“…In rows 8-10 and row 14, we compare the proposed Spatial Mask Decoder (SMD) with 3 alternatives. ( 1) Binary (row 9) [13]: a binary mask over the union feature to specify the position of [subject, object] ; (2) PE (row 10) [57]: naive positional embedding; (3) None (row 8): no spatial learning module. We conclude that: (1) adding spatial information learning generally improves the performance; (2) Implicitly imposing spatial information with PE or binary masks gives worse performance than SMD which enforces an explicit constraint over the feature space.…”

Section: Ablation Studiesmentioning

confidence: 99%

See 1 more Smart Citation

Towards Overcoming False Positives in Visual Relationship Detection

Jin,

Ma,

Zhang

et al. 2020

Preprint

View full text Add to dashboard Cite

In this paper, we investigate the cause of the high false positive rate in Visual Relationship Detection (VRD). We observe that during training, the relationship proposal distribution is highly imbalanced: most of the negative relationship proposals are easy to identify, e.g., the inaccurate object detection, which leads to the under-fitting of lowfrequency difficult proposals. This paper presents Spatially-Aware Balanced negative pRoposal sAmpling (SABRA), a robust VRD framework that alleviates the influence of false positives. To effectively optimize the model under imbalanced distribution, SABRA adopts Balanced Negative Proposal Sampling (BNPS) strategy for mini-batch sampling. BNPS divides proposals into 5 well defined sub-classes and generates a balanced training distribution according to the inverse frequency. BNPS gives an easier optimization landscape and significantly reduces the number of false positives. To further resolve the low-frequency challenging false positive proposals with high spatial ambiguity, we improve the spatial modeling ability of SABRA on two aspects: a simple and efficient multi-head heterogeneous graph attention network (MH-GAT) that models the global spatial interactions of objects, and a spatial mask decoder that learns the local spatial configuration. SABRA outperforms SOTA methods by a large margin on two human-object interaction (HOI) datasets and one general VRD dataset.

show abstract

Deeply Supervised Multimodal Attentional Translation Embeddings for Visual Relationship Detection

Cited by 12 publications

References 23 publications

Scene Graph Generation: A Comprehensive Survey

Scene Graph Generation: A Comprehensive Survey

Fully Convolutional Scene Graph Generation

Towards Overcoming False Positives in Visual Relationship Detection

Contact Info

Product

Resources

About