Improving Captioning for Low-Resource Languages by Cycle Consistency

Wu, Yike; Zhao, Shiwan; Jia, Chen; Zhang, Ying; Yuan, Xiaojie; Su, Zhong

doi:10.1109/icme.2019.00070

Cited by 8 publications

(4 citation statements)

References 19 publications

(30 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The two main strategies entail collecting captions in different languages for commonly used datasets (e.g. Chinese and Japanese captions for MS COCO images [200], [201], German captions for Flick30K [202]), or directly training multilingual captioning systems with unpaired captions [160], [199], [203], [204], [205], which requires specific evaluation protocols [206].…”

Section: Focusing On the Textual Outputmentioning

confidence: 99%

From Show to Tell: A Survey on Deep Learning-based Image Captioning

Stefanini¹,

Cornia²,

Baraldi³

et al. 2021

Preprint

View full text Add to dashboard Cite

Connecting Vision and Language plays an essential role in Generative Intelligence. For this reason, large research efforts have been devoted to image captioning, i.e. describing images with syntactically and semantically meaningful sentences. Starting from 2015 the task has generally been addressed with pipelines composed of a visual encoder and a language model for text generation. During these years, both components have evolved considerably through the exploitation of object regions, attributes, the introduction of multi-modal connections, fully-attentive approaches, and BERT-like early-fusion strategies. However, regardless of the impressive results, research in image captioning has not reached a conclusive answer yet. This work aims at providing a comprehensive overview of image captioning approaches, from visual encoding and text generation to training strategies, datasets, and evaluation metrics. In this respect, we quantitatively compare many relevant state-of-the-art approaches to identify the most impactful technical innovations in architectures and training strategies. Moreover, many variants of the problem and its open challenges are discussed. The final goal of this work is to serve as a tool for understanding the existing literature and highlighting the future directions for a research area where Computer Vision and Natural Language Processing can find an optimal synergy.

show abstract

Section: Focusing On the Textual Outputmentioning

confidence: 99%

From Show to Tell: A Survey on Deep Learning-based Image Captioning

Stefanini¹,

Cornia²,

Baraldi³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…To extend captioning technology to non-English languages, we are starting to see some studies being reported. Some researchers have attempted to directly propose a captioning model on a target language while utilizing a pivot language, typically English, in which paired information is readily avail-able [12,23,34,39]. Nevertheless, the straightforward approach remains to collect image-caption pairs in the target language (e.g., French [30], German [10], or Chinese [24]).…”

Section: Related Workmentioning

confidence: 99%

Object-Centric Unsupervised Image Captioning

Meng¹,

Yang²,

Cao³

et al. 2021

Preprint

View full text Add to dashboard Cite

“…Scene graphs [45], vmCAN [59], Graph-align [136], Know more say less [137], GCN-LSTM [15], SGAE [16], StructCap [138], GCH [139], GIN [140], Textual-GCNs [141], CSMN [31], CMMN [64], ReGAT [142], Out-of-the-box [143], Graph VQA [144], GERG [145], VKMN [146], MAN-VQA [147], DMN+ [148], MSCQA [116], SCH-GAN [82], CBT [149], SCST [17], CAVP [18], SR-PL [19], SMem-VQA [150], ODA [151], AOA [152], Up-Down [32], Attention-aware [79], BSSAN [62], CRAN [153], CBP [36], SOT [5], PAGNet [34], MirrorGAN [154], DAI [155], T2I2T [11], CCGAN [13], C4Synth [156], Cycle-Attn+ [157], Coupled CycleGAN [158], TCCM [159], CycleMatch [160], VQA-Rephrasings [161], iQAN [162], MLAN...…”

Section: Recent Advances In Deep Multimodal Feature Learningmentioning

confidence: 99%

New Ideas and Trends in Deep Multimodal Content Understanding: A Review

Chen

Wang²,

Liu

et al. 2020

Preprint

View full text Add to dashboard Cite

The focus of this survey is on the analysis of two modalities of multimodal deep learning: image and text. Unlike classic reviews of deep learning where monomodal image classifiers such as VGG, ResNet and Inception module are central topics, this paper will examine recent multimodal deep models and structures, including auto-encoders, generative adversarial nets and their variants. These models go beyond the simple image classifiers in which they can do uni-directional (e.g. image captioning, image generation) and bidirectional (e.g. cross-modal retrieval, visual question answering) multimodal tasks. Besides, we analyze two aspects of the challenge in terms of better content understanding in deep multimodal applications. We then introduce current ideas and trends in deep multimodal feature learning, such as feature embedding approaches and objective function design, which are crucial in overcoming the aforementioned challenges.Finally, we include several promising directions for future research.

show abstract

Improving Captioning for Low-Resource Languages by Cycle Consistency

Cited by 8 publications

References 19 publications

From Show to Tell: A Survey on Deep Learning-based Image Captioning

From Show to Tell: A Survey on Deep Learning-based Image Captioning

Object-Centric Unsupervised Image Captioning

New Ideas and Trends in Deep Multimodal Content Understanding: A Review

Contact Info

Product

Resources

About