Natural Language Object Retrieval

Hu, Ronghang; Xu, Huazhe; Rohrbach, Marcus; Feng, Jiashi; Saenko, Kate; Darrell, Trevor

doi:10.48550/arxiv.1511.04164

Cited by 12 publications

(15 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our work is related to recent work on object localization with natural language, where the task is to localize a target object in a scene from its natural language description (by drawing a bounding box over it). The methods reported in [11] and [13] build upon image captioning frameworks such as LRCN [14] or mRNN [15], and localize objects by selecting the bounding box where the expression has the highest probability. Our model differs from [11] and [13] in that we do not have to learn to generate expressions from image regions.…”

Section: Related Workmentioning

confidence: 99%

“…The methods reported in [11] and [13] build upon image captioning frameworks such as LRCN [14] or mRNN [15], and localize objects by selecting the bounding box where the expression has the highest probability. Our model differs from [11] and [13] in that we do not have to learn to generate expressions from image regions. In [12], the authors propose a model to localize a textual phrase by attending to a region on which the phrase can be best reconstructed.…”

Section: Related Workmentioning

confidence: 99%

“…To the best of our knowledge, all these previous localization methods can only return a bounding box of the target object, and no prior work has learned to directly output a segmentation mask of an object given a natural language description as query. As a comparison, in Section 4.1 we also evaluate using foreground segmentation over the bounding box prediction from [11] and [12].…”

Section: Related Workmentioning

confidence: 99%

“…Compared with related work [11,13], we do not explicitly produce a word sequence corresponding to object descriptions given a visual representation, since we are interested in predicting image segmentation from an expression rather than predicting the expression. In this way, our model has less parameters compared with [11,13] as it does not have to learn to predict the next word, which can be a hard task.…”

Section: Our Modelmentioning

confidence: 99%

“…As described in more details in Section 2, prior methods suitable for this task were limited to resolving only a bounding box in the image [11,12,13], and/or were limited to a fixed set of categories determined a priori [1,3,4,6]. In this paper, we propose an end-to-end trainable recurrent convolutional network model that jointly learns to process visual and linguistic information, and produces segmentation output for the target image region described by the natural language expression, as illustrated in Figure 2.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Segmentation from Natural Language Expressions

Rohrbach

Darrell

2016

Lecture Notes in Computer Science

Self Cite

278

315

View full text Add to dashboard Cite

In this paper we approach the novel problem of segmenting an image based on a natural language expression. This is different from traditional semantic segmentation over a predefined set of semantic classes, as e.g., the phrase "two men sitting on the right bench" requires segmenting only the two people on the right bench and no one standing or sitting on another bench. Previous approaches suitable for this task were limited to a fixed set of categories and/or rectangular regions. To produce pixelwise segmentation for the language expression, we propose an end-to-end trainable recurrent and convolutional network model that jointly learns to process visual and linguistic information. In our model, a recurrent LSTM network is used to encode the referential expression into a vector representation, and a fully convolutional network is used to a extract a spatial feature map from the image and output a spatial response map for the target object. We demonstrate on a benchmark dataset that our model can produce quality segmentation output from the natural language expression, and outperforms baseline methods by a large margin.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Our Modelmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Segmentation from Natural Language Expressions

Rohrbach

Darrell

2016

Lecture Notes in Computer Science

Self Cite

278

315

View full text Add to dashboard Cite

show abstract

Attention Correctness in Neural Image Captioning

Liu¹

2017

AAAI

113

View full text Add to dashboard Cite

Attention mechanisms have recently been introduced in deep learning for various tasks in natural language processing and computer vision. But despite their popularity, the ``correctness'' of the implicitly-learned attention maps has only been assessed qualitatively by visualization of several examples. In this paper we focus on evaluating and improving the correctness of attention in neural image captioning models. Specifically, we propose a quantitative evaluation metric for the consistency between the generated attention maps and human annotations, using recently released datasets with alignment between regions in images and entities in captions. We then propose novel models with different levels of explicit supervision for learning attention maps during training. The supervision can be strong when alignment between regions and caption entities are available, or weak when only object segments and categories are provided. We show on the popular Flickr30k and COCO datasets that introducing supervision of attention maps during training solidly improves both attention correctness and caption quality, showing the promise of making machine perception more human-like.

show abstract

Segmentation from Natural Language Expressions

Rohrbach

Darrell

2016

Preprint

View full text Add to dashboard Cite

Natural Language Object Retrieval

Cited by 12 publications

References 16 publications

Segmentation from Natural Language Expressions

Segmentation from Natural Language Expressions

Attention Correctness in Neural Image Captioning

Segmentation from Natural Language Expressions

Contact Info

Product

Resources

About