Visual Translation Embedding Network for Visual Relation Detection

Zhang, Hanwang; Kyaw, Zawlin; Chang, Shih-Fu; Chua, Tat-Seng

doi:10.48550/arxiv.1702.08319

Cited by 23 publications

(2 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our work also has similarities to knowledge bases, where predicates are often projections in a defined semantic space [3,6,22]. Such a method was recently used for visual relationship detection [43]. While these methods have seen success in knowledge base completion tasks, they have only led to a marginal gain for modelling visual relationships.…”

Section: Related Workmentioning

confidence: 99%

Referring Relationships

Krishna

Chami

Bernstein

et al. 2018

2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition

View full text Add to dashboard Cite

Images are not simply sets of objects: each image represents a web of interconnected relationships. These relationships between entities carry semantic meaning and help a viewer differentiate between instances of an entity. For example, in an image of a soccer match, there may be multiple persons present, but each participates in different relationships: one is kicking the ball, and the other is guarding the goal. In this paper, we formulate the task of utilizing these "referring relationships" to disambiguate between entities of the same category. We introduce an iterative model that localizes the two entities in the referring relationship, conditioned on one another. We formulate the cyclic condition between the entities in a relationship by modelling predicates that connect the entities as shifts in attention from one entity to another. We demonstrate that our model can not only outperform existing approaches on three datasets -CLEVR, VRD and Visual Genome -but also that it produces visually meaningful predicate shifts, as an instance of interpretable neural networks. Finally, we show that by modelling predicates as attention shifts, we can even localize entities in the absence of their category, allowing our model to find completely unseen categories.

show abstract

Section: Related Workmentioning

confidence: 99%

Referring Relationships

Krishna

Chami

Bernstein

et al. 2018

2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition

View full text Add to dashboard Cite

show abstract

“…An alternative is to list all important objects with their attributes and relationships. Johnson et al [17] created scene graphs, which are being used for visual relationship detection [27,30,49] tasks. In [25], the authors exploit scene graphs to generate image captions.…”

Section: Related Workmentioning

confidence: 99%

Situation Recognition with Graph Neural Networks

Tapaswi

Liao

et al. 2017

2017 IEEE International Conference on Computer Vision (ICCV)

113

View full text Add to dashboard Cite

We address the problem of recognizing situations in images. Given an image, the task is to predict the most salient verb (action), and fill its semantic roles such as who is performing the action, what is the source and target of the action, etc. Different verbs have different roles (e.g. attacking has weapon), and each role can take on many possible values (nouns). We propose a model based on Graph Neural Networks that allows us to efficiently capture joint dependencies between roles using neural networks defined on a graph. Experiments with different graph connectivities show that our approach that propagates information between roles significantly outperforms existing work, as well as multiple baselines. We obtain roughly 3-5% improvement over previous work in predicting the full situation. We also provide a thorough qualitative analysis of our model and influence of different roles in the verbs. VERB: REPAIRING PROBLEM Part Pipe Wheel ⋯ ITEM AC Wall Car ⋯ TOOL Screwdriver Wrench Knife ⋯ PLACE Outdoors Bathroom Field

show abstract

Multimodal Latent Factor Model with Language Constraint for Predicate Detection

Yao

2019

2019 IEEE International Conference on Image Processing (ICIP)

View full text Add to dashboard Cite

Visual Translation Embedding Network for Visual Relation Detection

Cited by 23 publications

References 9 publications

Referring Relationships

Referring Relationships

Situation Recognition with Graph Neural Networks

Multimodal Latent Factor Model with Language Constraint for Predicate Detection

Contact Info

Product

Resources

About