Describing and Localizing Multiple Changes with Transformers

Abstract

Change captioning tasks aim to detect changes in image pairs observed before and after a scene change and generate a natural language description of the changes. Existing change captioning studies have mainly focused on scenes with a single change. However, detecting and describing multiple changed parts in image pairs is essential for enhancing adaptability to complex scenarios. We solve the above issues from three aspects: (i) We propose a CG-based multi-change captioning dataset; (ii) We benchmark existing state-of-the-art methods of single change captioning on multi-change captioning; (iii) We further propose Multi-Change Captioning transformers (MCCFormers) that identify change regions by densely correlating different regions in image pairs and dynamically determines the related change regions with words in sentences. The proposed method obtained the highest scores on four conventional change captioning evaluation metrics for multichange captioning. In addition, existing methods generate a single attention map for multiple changes and lack the ability to distinguish change regions. In contrast, our proposed method can separate attention maps for each change and performs well with respect to change localization. Moreover, the proposed framework outperformed the previous state-of-the-art methods on an existing change captioning benchmark, CLEVR-Change, by a large margin (+6.1 on BLEU-4 and +9.7 on CIDEr scores), indicating its general ability in change captioning tasks. Our code and dataset will be publicly available through the project page 1 . Before Change captions After Caption 1: The large gray rubber sphere has disappeared. (delete) Caption 2: There is no longer a large cyan metal cube. (delete) Caption 3: The large brown metal sphere was moved from its original location. (move) Caption 4: The small yellow rubber cylinder was replaced by a small red rubber sphere. (replace)

Expert Knowledge-Aware Image Difference Graph Representation Learning for Difference-Aware Medical Visual Question Answering

“…To date, no prior research has specifically addressed the "image difference question answering" problem. Only a few studies have focused on the general image difference caption task, such as MMCFormers [25] and IDCPCL [31]. Therefore, our work serves as the first step in this new direction and provides a valuable contribution to the research community.…”

“…Within the language generation and vision research domain, the most related works to the medical image difference VQA task is image difference captioning [20,25,31], which is designed to identify object movements and changes within a spatial context such as a static or complex background. As shown in the left Fig.…”

“…2.MCCFormers is proposed to handle the image difference captioning task [25]. It achieved state-of-the-art performance on the CLEVR-Change dataset [22], a famous image difference captioning dataset.…”

The Change You Want to See

“…Most of the existing works in the change detection literature belong to this category. [18,17,21,12] tackle the change captioning problem where the goal is to describe the changes in an image pair in natural language. These methods mainly evaluate their approach on the STD [10] (images from fixed video surveillance camera), or CLEVR-based change datasets [18,21,12] (synthetic images of 3D objects of primitive shapes).…”

Rethinking the Reference-based Distinctive Image Captioning

“…According to the number of input images, they can be divided into two categories: Two-image based and Groupbased captioning. Two-image based captioning tends to describe the common [36] or different [28,30,37,49] parts between the two images. Thus, the two images in their settings always have strong correlations.…”