Reinforcing an Image Caption Generator Using Off-Line Human Feedback

Seo, Paul Hongsuck; Sharma, Piyush; Levinboim, Tomer; Han, Bohyung; Soricut, Radu

doi:10.1609/aaai.v34i03.5655

Cited by 17 publications

(4 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For example, Neural Image Captioning (NIC) [4] treated the image captioning task as a translation problem from vision to text, and it was the first image captioning model that exploited the encoder-decoder paradigm. After that, the attention mechanisms [7,21,22], the training strategy with reinforcement learning [23][24][25], and the large-scale pre-trained visionlanguage models [26,27] enrich the captioning methods based on deep learning.…”

Section: Image Captioningmentioning

confidence: 99%

Exploring better image captioning with grid features

Yan,

Xie,

Guo

et al. 2024

Complex Intell. Syst.

View full text Add to dashboard Cite

Nowadays, Artificial Intelligence Generated Content (AIGC) has shown promising prospects in both computer vision and natural language processing communities. Meanwhile, as an essential aspect of AIGC, image to captions has received much more attention. Recent vision-language research is developing from the bulky region visual representations based on object detectors toward more convenient and flexible grid ones. However, this kind of research typically concentrates on image understanding tasks like image classification, with less attention paid to content generation tasks. In this paper, we explore how to capitalize on the expressive features embedded in the grid visual representations for better image captioning. To this end, we present a Transformer-based image captioning model, dubbed FeiM, with two straightforward yet effective designs. We first design the feature queries that consist of a limited set of learnable vectors, which act as the local signals to capture specific visual information from global grid features. Then, taking augmented global grid features and the local feature queries as inputs, we develop a feature interaction module to query relevant visual concepts from grid features, and to enable interaction between the local signal and overall context. Finally, the refined grid visual representations and the linguistic features pass through a Transformer architecture for multi-modal fusion. With the two novel and simple designs, FeiM can fully leverage meaningful visual knowledge to improve image captioning performance. Extensive experiments are performed on the competitive MSCOCO benchmark to confirm the effectiveness of the proposed approach, and the results show that FeiM yields more eminent results than existing advanced captioning models.

show abstract

Section: Image Captioningmentioning

confidence: 99%

Exploring better image captioning with grid features

Yan,

Xie,

Guo

et al. 2024

Complex Intell. Syst.

View full text Add to dashboard Cite

show abstract

“…The easy-to-hard spirit of CL (Bengio et al, 2009) is inspired by the learning process of us humans. This human-like effective training paradigm has been extensively exploited in different vision-language tasks Seo et al, 2020;Zheng et al, 2022;Yao et al, 2021). Recent works (Lao et al, 2021;Pan et al, 2022) tried to introduce CL into VQA to reduce language biases by helping VQA models gradually focus on more biased samples (Lao et al, 2021) or gradually increasing the importance of visual features in the training phase (Pan et al, 2022).…”

Section: Related Workmentioning

confidence: 99%

Rethinking Data Augmentation for Robust Visual Question Answering

Chen¹,

Zheng²,

Xiao³

2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Being widely used in learning unbiased visual question answering (VQA) models, Data Augmentation (DA) helps mitigate language biases by generating extra training samples beyond the original samples. While today's DA methods can generate robust samples, the augmented training set, significantly larger than the original dataset, often exhibits redundancy in terms of difficulty or content repetition, leading to inefficient model training and even compromising the model performance. To this end, we design an Effective Curriculum Learning strategy ECL to enhance DA-based VQA methods. Intuitively, ECL trains VQA models on relatively "easy" samples first, and then gradually changes to "harder" samples, and less-valuable samples are dynamically removed. Compared to training on the entire augmented dataset, our ECL strategy can further enhance VQA models' performance with fewer training samples. Extensive ablations have demonstrated the effectiveness of ECL on various methods.

show abstract

“…Existing image captioning approaches typically follow the encoder-decoder architecture (Xu et al 2015;Huang et al 2019;Guo et al 2020;Cornia et al 2020;Zhao, Wu, and Zhang 2020;Seo et al 2020), which takes an image as input and generates a description in the form of natural language. Earlier works (Xu et al 2015;Lu et al 2017;Jiang et al 2020) apply grid-based features as input to generate captions, which are fixed-size patches extracted from the CNN (He et al 2016; ?)…”

Section: Related Workmentioning

confidence: 99%

Dual-level Collaborative Transformer for Image Captioning

Luo

Sun

et al. 2021

AAAI

188

View full text Add to dashboard Cite

Descriptive region features extracted by object detection networks have played an important role in the recent advancements of image captioning. However, they are still criticized for the lack of contextual information and fine-grained details, which in contrast are the merits of traditional grid features. In this paper, we introduce a novel Dual-Level Collaborative Transformer (DLCT) network to realize the complementary advantages of the two features. Concretely, in DLCT, these two features are first processed by a novel Dual-way Self Attenion (DWSA) to mine their intrinsic properties, where a Comprehensive Relation Attention component is also introduced to embed the geometric information. In addition, we propose a Locality-Constrained Cross Attention module to address the semantic noises caused by the direct fusion of these two features, where a geometric alignment graph is constructed to accurately align and reinforce region and grid features. To validate our model, we conduct extensive experiments on the highly competitive MS-COCO dataset, and achieve new state-of-the-art performance on both local and online test sets, i.e., 133.8% CIDEr on Karpathy split and 135.4% CIDEr on the official split.

show abstract

Reinforcing an Image Caption Generator Using Off-Line Human Feedback

Cited by 17 publications

References 15 publications

Exploring better image captioning with grid features

Exploring better image captioning with grid features

Rethinking Data Augmentation for Robust Visual Question Answering

Dual-level Collaborative Transformer for Image Captioning

Contact Info

Product

Resources

About