Group-based Distinctive Image Captioning with Memory Attention

Abstract

Describing images using natural language is widely known as image captioning, which has made consistent progress due to the development of computer vision and natural language generation techniques. Though conventional captioning models achieve high accuracy based on popular metrics, i.e., BLEU, CIDEr, and SPICE, the ability of captions to distinguish the target image from other similar images is under-explored. To generate distinctive captions, a few pioneers employ contrastive learning or re-weighted the ground-truth captions, which focuses on one single input image. However, the relationships between objects in a similar image group (e.g., items or properties within the same album or fine-grained events) are neglected. In this paper, we improve the distinctiveness of image captions using a Group-based Distinctive Captioning Model (GdisCap), which compares each image with other images in one similar group and highlights the uniqueness of each image. In particular, we propose a group-based memory attention (GMA) module, which stores object features that are unique among the image group (i.e., with low similarity to objects in other images). These unique object features are highlighted when generating captions, resulting in more distinctive captions. Furthermore, the distinctive words in the ground-truth captions are selected to supervise the language decoder and GMA. Finally, we propose a new evaluation metric, distinctive word rate (DisWordRate) to measure the distinctiveness of captions. Quantitative results indicate that the proposed method significantly improves the distinctiveness of several baseline models, and achieves the state-of-the-art performance on both accuracy and distinctiveness. Results of a user study agree with the quantitative evaluation and demonstrate the rationality of the new metric DisWordRate. CCS CONCEPTS• Computing methodologies → Computer vision.

A thorough review of models, evaluation metrics, and datasets on image captioning

“…Wang et al. [59] improved the distinctiveness of image captions using a group‐based distinctive captioning model, and proposed a new evaluation metric DisWordRate to measure the distinctiveness of captions.…”

Deep Vision Multimodal Learning: Methodology, Benchmark, and Trend

“…The multiperspective captioning task [236] considers text and image inputs from different viewpoints. The distinctive image captioning task [237] is defined to describe the unique object or context of an image to distinguish it from other semantically similar images. The diverse caption and rich image generation task [238] proposes a bidirectional image and text generation task to align rich images and their corresponding multiple different titles, aiming to achieve multiple sentences from one image uniformly and multiple sentences to generate more suitable images.…”

Rethinking the Reference-based Distinctive Image Captioning

“…For human-like distinctive captioning, a recent work proposes to study the DIC task based on a group of semantic-similar reference images, dubbed Reference-based DIC (Ref-DIC). Different [43]. (b): Selected reference images for the same target image using our two-stage matching mechanism.…”

“…Compared to Single-DIC, the generated captions are only asked to distinguish the target image from the group of reference images, i.e., grouplevel distinctiveness. Unfortunately, the reference images used in existing Ref-DIC works [43] can be trivially distinguished: these reference images only resemble the target image at the scene-level and have few common objects, thus Ref-DIC models can simply generate distinctive captions even without considering the reference images. For example in Figure 1 (a), target and reference images have no object in common (e.g., "towel", "shower curtain", or "toilet"), each object in target image is unique, such that the Ref-DIC model can trivially generate "a bathroom with a towel" to tell the target and reference images apart.…”

“…In this paper, we propose two new benchmarks for the Ref-DIC task: COCO-DIC and Flickr30K-DIC. To strictly control the unique details between target and reference images, we propose a two-stage matching mechanism, which can measure image similarity at the object-/attribute-level (vs. scene-level in [43]), and deliberately make target and reference images have some common objects. Under this mechanism, Ref-DIC models can learn to focus on the unique attributes and objects in target image.…”