Rich Image Captioning in the Wild

Tran, Kenneth; He, Xiaodong; Zhang, Lei; Sun, Jian

doi:10.1109/cvprw.2016.61

Cited by 100 publications

(64 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recent progress in image captioning, the task of generating natural language descriptions of visual content [11,12,18,19,43,46], can be largely attributed to the publicly available large-scale datasets of image-caption pairs [6,16,50] as well as steady modeling improvements [4,26,37,48]. However, these models generalize poorly to images in the wild [39] despite impressive benchmark performance, because they are trained on datasets which cover a tiny fraction of the long-tailed distribution of visual concepts in the real world. For example, models trained on COCO Captions [6] can typically describe images containing dogs, people and umbrellas, but not accordions or dolphins.…”

Section: Introductionmentioning

confidence: 99%

nocaps: novel object captioning at scale

Agrawal

Anderson²,

Desai³

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

185

157

View full text Add to dashboard Cite

show abstract

Section: Introductionmentioning

confidence: 99%

nocaps: novel object captioning at scale

Agrawal

Anderson²,

Desai³

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

185

157

View full text Add to dashboard Cite

show abstract

“…also Section 2). Recently, the captionbot system [10] was proposed to generate captions for a given image. However, all these approaches focus on the description of what is to be explicitly found, i.e., depictable, within pictures.…”

Section: Introductionmentioning

confidence: 99%

Knowledge-rich image gist understanding beyond literal meaning

Weiland

Hulpuș

Ponzetto

et al. 2018

Data & Knowledge Engineering

View full text Add to dashboard Cite

We investigate the problem of understanding the message (gist) conveyed by images and their captions as found, for instance, on websites or news articles. To this end, we propose a methodology to capture the meaning of image-caption pairs on the basis of large amounts of machine-readable knowledge that has previously been shown to be highly effective for text understanding. Our method identifies the connotation of objects beyond their denotation: where most approaches to image understanding focus on the denotation of objects, i.e., their literal meaning, our work addresses the identification of connotations, i.e., iconic meanings of objects, to understand the message of images. We view image understanding as the task of representing an image-caption pair on the basis of a wide-coverage vocabulary of concepts such as the one provided by Wikipedia, and cast gist detection as a concept-ranking problem with image-caption pairs as queries. Our proposed algorithm brings together aspects of entity linking and clustering, subgraph selection, semantic relatedness, and learning-to-rank in a novel way. In addition to this novel task and a complete evaluation of our approach, we introduce a novel dataset to foster further research on this problem. To enable a thorough investigation of the problem of gist understanding, we produce a gold standard of over 300 image-caption pairs and over 8,000 gist annotations covering a wide variety of topics at different levels of abstraction. We use this dataset to experimentally benchmark the contribution of signals from heterogeneous sources, namely image and text. The best result with a Mean Average Precision (MAP) of 0.69 indicate that by combining both dimensions we are able to better understand the meaning of our image-caption pairs than when using language or vision information alone. We test the robustness of our gist detection approach when receiving automatically generated input, i.e., using automatically generated image tags or generated captions, and prove the feasibility of an end-to-end automated process. However, we also show experimentally that stateof-the-art image and text understanding is better at dealing with literal meanings of image-caption pairs, with non-literal pairs being instead generally more difficult to detect, thus paving the way for future work on understanding the message of images beyond their literal content.

show abstract

“…It takes the previously generated words and then finds a sentence/caption that has the highest likelihood to caption the image that contains every word it has detected. Tran et al [17] have produced a system that could richly caption images. Their research claims to be able to detect and classify a large range of visual concepts.…”

Section: Recent Research On Image Captioningmentioning

confidence: 99%

“…Tran et al [17] Rich description -Adding specifics to image, such as person and location  Presenting a caption model for open domain images, which utilizes a composite approach.  Enriching existing frameworks with visual concepts such as landmarks and celebrity identification.…”

Section: Related Workmentioning

confidence: 99%