Conditional Image-Text Embedding Networks

Plummer, Bryan A.; Kordas, Paige; Kiapour, M. Hadi; Zheng, Shuai; Piramuthu, Robinson; Lazebnik, Svetlana

doi:10.1007/978-3-030-01258-8_16

Cited by 93 publications

(54 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For a fair comparison, all these methods use a fixed RPN to obtain the candidate boxes and represent them in features that are not tuned on the Flickr30K Entities dataset. We believe that using an additional conditional embedding unit as in [32], and the integration of a proposal generation network with a spatial regression that is tuned on Flickr30K Entities as in [3] should improve the overall result even more. Table 2 shows the phrase grounding performance with respect to the coarse categories in Flickr30K Entitites dataset.…”

Section: Resultsmentioning

confidence: 99%

“…Wang et al [48] proposes a structured matching method which attempts to reflect the semantic relation of phrases onto the visual relations of their corresponding regions without considering the global sentence-level context. Plummer et al [32] proposes to use multiple text-conditioned embeddings in a single end-to-end model with impressive results on Flickr30K Entities dataset [34].…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Neural Sequential Phrase Grounding (SeqGROUND)

Dogan

Sigal

Groß

2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

View full text Add to dashboard Cite

We propose an end-to-end approach for phrase grounding in images. Unlike prior methods that typically attempt to ground each phrase independently by building an imagetext embedding, our architecture formulates grounding of multiple phrases as a sequential and contextual process. Specifically, we encode region proposals and all phrases into two stacks of LSTM cells, along with so-far grounded phrase-region pairs. These LSTM stacks collectively capture context for grounding of the next phrase. The resulting architecture, which we call SeqGROUND, supports many-to-many matching by allowing an image region to be matched to multiple phrases and vice versa. We show competitive performance on the Flickr30K benchmark dataset and, through ablation studies, validate the efficacy of sequential grounding as well as individual design choices in our model architecture.

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Neural Sequential Phrase Grounding (SeqGROUND)

Dogan

Sigal

Groß

2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

View full text Add to dashboard Cite

show abstract

“…Accu@0.5 SCRC [19] 17.93 MCB + Reg + Spatial [5] 26.54 GroundeR + Spatial [43] 26.93 Similarity Network + Spatial [47] 31.26 CGRE [37] 31.85 MNN + Reg + Spatial [5] 32.21 EB + QRN (VGG cls -SPAT) [6] 32.21 CITE [38] 34.13 IGOP [52] 34.70 QRC Net [6] 44.07 G3RAPHGROUND++ [3] 44.91…”

Section: Methodsmentioning

confidence: 99%

“…We validate our model with different alignment regularization weights and find 3 gives the best performance, so is set to 3 in all following experiments. [6] 53.48 IGOP [52] 53.97 SPC+PPC [39] 55.49 SS+QRN [6] 55.99 CITE [38] 59.27 SeqGROUND [10] 61.60 G3RAPHGROUND++ [3] 66.93 Visual-BERT [30] 71.33 Contextual Grounding [29] 71.36…”

Section: Implementation Detailsmentioning

confidence: 99%

Cross-Modal Omni Interaction Modeling for Phrase Grounding

Hui

et al. 2020

Proceedings of the 28th ACM International Conference on Multimedia

View full text Add to dashboard Cite

Phrase grounding aims to localize the objects described by phrases in a natural language specification. Previous works model the interaction of inputs from text modality and visual modality only in the intra-modal global level and consequently lacks the ability to capture the precise and complete context information. In this paper, we propose a novel Cross-Modal Omni Interaction network (COI Net) composed of a neighboring interaction module, a global interaction module, a cross-modal interaction module and a multilevel alignment module. Our approach formulates the complex spatial and semantic relationship among image regions and phrases through multi-level multi-modal interaction. We capture the local relationship using the interaction among neighboring regions and then collect the global context through the interaction among all regions using a transformer encoder. We further use a co-attention module to apply the interaction between two modalities to gather the crossmodal context for all image regions and phrases. In addition to the omni interaction modeling, we also leverage a straightforward yet effective multilevel alignment regularization to formulate the dependencies among all grounding decisions. We extensively validate the effectiveness of our model. Experiments show that our approach outperforms existing state-of-the-art methods by large margins on two popular datasets in terms of accuracy: 6.15% on Flickr30K Entities (71.36% increased to 77.51%) and 21.25% on ReferItGame (44.91% * Equal contribution.

show abstract

“…Rohrbach et al (2016) proposed an attention mechanism to attend to relevant object proposals for a given phrase and designed a loss for phrase reconstruction. Plummer et al (2018) presented an approach to jointly learn multiple text-conditioned embedding in a single endto-end network. In DDPN (Yu et al 2018b), they learned a diversified and discriminate proposal network to generate higher quality object candidates.…”

Section: Related Workmentioning

confidence: 99%

Learning Cross-Modal Context Graph for Visual Grounding

Liu

Wan

Zhu

et al. 2020

AAAI

View full text Add to dashboard Cite

Visual grounding is a ubiquitous building block in many vision-language tasks and yet remains challenging due to large variations in visual and linguistic features of grounding entities, strong context effect and the resulting semantic ambiguities. Prior works typically focus on learning representations of individual phrases with limited context information. To address their limitations, this paper proposes a language-guided graph representation to capture the global context of grounding entities and their relations, and develop a cross-modal graph matching strategy for the multiple-phrase visual grounding task. In particular, we introduce a modular graph neural network to compute context-aware representations of phrases and object proposals respectively via message propagation, followed by a graph-based matching module to generate globally consistent localization of grounding phrases. We train the entire graph neural network jointly in a two-stage strategy and evaluate it on the Flickr30K Entities benchmark. Extensive experiments show that our method outperforms the prior state of the arts by a sizable margin, evidencing the efficacy of our grounding framework. Code is available at https://github.com/youngfly11/LCMCG-PyTorch.

show abstract

Conditional Image-Text Embedding Networks

Cited by 93 publications

References 36 publications

Neural Sequential Phrase Grounding (SeqGROUND)

Neural Sequential Phrase Grounding (SeqGROUND)

Cross-Modal Omni Interaction Modeling for Phrase Grounding

Learning Cross-Modal Context Graph for Visual Grounding

Contact Info

Product

Resources

About