Dynamic Multimodal Instance Segmentation Guided by Natural Language Queries

Margffoy-Tuay, Edgar A.; Pérez, Juan C.; Botero, Emilio; Arbeláez, Pablo

doi:10.1007/978-3-030-01252-6_39

Cited by 143 publications

(95 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We contrast our approach to some closely related works, including two existing one-stage grounding methods [52,44] and some on grounding as segmentation [13,20,25,9].…”

Section: Comparison To Other One-stage Grounding Workmentioning

confidence: 99%

A Fast and Accurate One-Stage Approach to Visual Grounding

Yang

Gong

Wang

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

300

286

View full text Add to dashboard Cite

We propose a simple, fast, and accurate one-stage approach to visual grounding, inspired by the following insight. The performances of existing propose-and-rank twostage methods are capped by the quality of the region candidates they propose in the first stage -if none of the candidates could cover the ground truth region, there is no hope in the second stage to rank the right region to the top. To avoid this caveat, we propose a one-stage model that enables end-to-end joint optimization. The main idea is as straightforward as fusing a text query's embedding into the YOLOv3 object detector, augmented by spatial features so as to account for spatial mentions in the query. Despite being simple, this one-stage approach shows great potential in terms of both accuracy and speed for both phrase localization and referring expression comprehension, according to our experiments. Given these results along with careful investigations into some popular region proposals, we advocate for visual grounding a paradigm shift from the conventional two-stage methods to the one-stage framework.

show abstract

“…We contrast our approach to some closely related works, including two existing one-stage grounding methods [52,44] and some on grounding as segmentation [13,20,25,9].…”

Section: Comparison To Other One-stage Grounding Workmentioning

confidence: 99%

A Fast and Accurate One-Stage Approach to Visual Grounding

Yang

Gong

Wang

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

300

286

View full text Add to dashboard Cite

show abstract

“…To better achieve word-to-image interaction, [17] directly combines visual features with each word feature from a language LSTM to recurrently refine segmentation results. Dynamic filter [20] for each word further enhances this interaction. In [22], word attention is incorporated in the image regions to model key-word-aware context.…”

Section: Related Workmentioning

confidence: 99%

“…Secondly, some previous works (e.g. [17,20]) process each word in the referring expression and concatenate it with visual features to infer the referred object in a sequential order using a recurrent network. The limitation is that these methods only look at local spatial regions and lack the interaction over long-range spatial regions in global context which is essential for semantic understanding and segmentation.…”

Section: Our Modelmentioning

confidence: 99%

Cross-Modal Self-Attention Network for Referring Image Segmentation

Rochan

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

382

237

View full text Add to dashboard Cite

We consider the problem of referring image segmentation. Given an input image and a natural language expression, the goal is to segment the object referred by the language expression in the image. Existing works in this area treat the language expression and the input image separately in their representations. They do not sufficiently capture long-range correlations between these two modalities. In this paper, we propose a cross-modal self-attention (CMSA) module that effectively captures the long-range dependencies between linguistic and visual features. Our model can adaptively focus on informative words in the referring expression and important regions in the input image. In addition, we propose a gated multi-level fusion module to selectively integrate self-attentive cross-modal features corresponding to different levels in the image. This module controls the information flow of features at different levels. We validate the proposed approach on four evaluation datasets. Our proposed approach consistently outperforms existing state-of-the-art methods.

show abstract

“…[21] used a convolutional LSTM in addition to the language-only LSTM to facilitate propagation of intermediate segmentation beliefs. [20,26] improved upon [21] by making more architectural improvements.…”

Section: Referring Expressionsmentioning

confidence: 99%

CLEVR-Ref+: Diagnosing Visual Reasoning With Referring Expressions

Liu

Bai

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

View full text Add to dashboard Cite

Referring object detection and referring image segmentation are important tasks that require joint understanding of visual information and natural language. Yet there has been evidence that current benchmark datasets suffer from bias, and current state-of-the-art models cannot be easily evaluated on their intermediate reasoning process. To address these issues and complement similar efforts in visual question answering, we build CLEVR-Ref+, a synthetic diagnostic dataset for referring expression comprehension. The precise locations and attributes of the objects are readily available, and the referring expressions are automatically associated with functional programs. The synthetic nature allows control over dataset bias (through sampling strategy), and the modular programs enable intermediate reasoning ground truth without human annotators.In addition to evaluating several state-of-the-art models on CLEVR-Ref+, we also propose IEP-Ref, a module network approach that significantly outperforms other models on our dataset. In particular, we present two interesting and important findings using IEP-Ref:(1) the module trained to transform feature maps into segmentation masks can be attached to any intermediate module to reveal the entire reasoning process step-by-step; (2) even if all training data has at least one object referred, IEP-Ref can correctly predict no-foreground when presented with false-premise referring expressions. To the best of our knowledge, this is the first direct and quantitative proof that neural modules behave in the way they are intended. 1

show abstract

Dynamic Multimodal Instance Segmentation Guided by Natural Language Queries

Cited by 143 publications

References 23 publications

A Fast and Accurate One-Stage Approach to Visual Grounding

A Fast and Accurate One-Stage Approach to Visual Grounding

Cross-Modal Self-Attention Network for Referring Image Segmentation

CLEVR-Ref+: Diagnosing Visual Reasoning With Referring Expressions

Contact Info

Product

Resources

About