Efficient Modeling of Future Context for Image Captioning

Abstract

Existing approaches to image captioning usually generate the sentence word-by-word from left to right, with the constraint of conditioned on local context including the given image and history generated words. There have been many studies target to make use of global information during decoding, e.g., iterative refinement. However, it is still under-explored how to effectively and efficiently incorporate the future context. To respond to this issue, inspired by that Non-Autoregressive Image Captioning (NAIC) can leverage two-side relation with modified mask operation, we aim to graft this advance to the conventional Autoregressive Image Captioning (AIC) model while maintaining the inference efficiency without extra time cost. Specifically, AIC and NAIC models are first trained combined with shared visual encoders, forcing the visual encoder to contain sufficient and valid future context; then the AIC model is encouraged to capture the causal dynamics of cross-layer interchanging from NAIC model on its unconfident words, which follows a teacher-student paradigm and optimized with the distribution calibration training objective. Empirical evidences demonstrate that our proposed approach clearly surpass the state-of-the-art baselines in both automatic metrics and human evaluations on the MS COCO benchmark. The source code is available at: https://github.com/feizc/Future-Caption. CCS CONCEPTS• Computing methodologies → Computer vision; Natural language processing.

Supervised Deep Learning Techniques for Image Description: A Systematic Review

“…This gives the resulting sentences more detail in their descriptions of the scene than traditional approaches. More recently, Fei [ 67 ] proposed a model that generates descriptions that effectively exploit the global context of the scene without implying an additional cost of inference. The model is trained with two sets: one contains the description labels, and the other includes the description of the general context of the image.…”

Improving Panoptic Narrative Grounding by Harnessing Semantic Relationships and Visual Confirmation

“…Recently, the growing interest in multimodal research (Fei 2022;Li et al 2022a;Chen et al 2022;Jing et al 2020;Ma et al 2022Ma et al , 2023Ji et al 2022;Huang et al 2023;Zhao et al 2023;Wu et al 2023) at the intersection of computer vision and natural language processing has driven the development of systems that can understand and describe the world as humans do. Panoptic Narrative Grounding (PNG) (González et al 2021) is an emerging visuallygrounded language understanding task that aims to locate and segment all instances of objects and regions in an image, corresponding to a given text description using binary pixel masks.…”