3D-Aware Scene Change Captioning From Multiview Images

Qiu, Yue; Satoh, Yutaka; Suzuki, Ryoichi; Iwata, Kenji; Kataoka, Hirokatsu

doi:10.1109/lra.2020.3003290

Cited by 23 publications

(13 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In these datasets, the states of the scenes are static. Qiu et al [ 35 ] proposed a simulated dataset for scene change captioning from multiview RGB images. However, they generated scenes with a solid background color, limiting the visual complexity.…”

Section: Related Workmentioning

confidence: 99%

“…Several recent works have discussed captioning based on images that include scene changes [ 33 , 34 , 35 ]. Difference Description with Latent Alignment (DDLA) [ 33 ] generates change descriptions from two video frames observed from different time steps of a given scene.…”

Section: Related Workmentioning

confidence: 99%

“…However, both DDLA and DUDA neglect the 3D geometry information of scenes and thus are less suitable for scenes with occlusions. Qiu et al [ 35 ] proposed a method that generates a compact scene representation from multiview images and then generates captions based on the scene representation. However, they performed experiments using scenes with solid colored backgrounds, and only considered RGB images from a fixed number of cameras.…”

Section: Related Workmentioning

confidence: 99%

“…However, they use single-view image inputs and neglect the geometric information of a scene, which limits their capability in scenes that contain occlusions. Qiu et al [ 35 ] proposed scene change captioning based on multiview observations made before and after a scene change. However, they considered only simulated table scenes with limited visual complexity.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Indoor Scene Change Captioning Based on Multimodality Data

Qiu

Satoh

Suzuki

et al. 2020

Sensors

Self Cite

View full text Add to dashboard Cite

This study proposes a framework for describing a scene change using natural language text based on indoor scene observations conducted before and after a scene change. The recognition of scene changes plays an essential role in a variety of real-world applications, such as scene anomaly detection. Most scene understanding research has focused on static scenes. Most existing scene change captioning methods detect scene changes from single-view RGB images, neglecting the underlying three-dimensional structures. Previous three-dimensional scene change captioning methods use simulated scenes consisting of geometry primitives, making it unsuitable for real-world applications. To solve these problems, we automatically generated large-scale indoor scene change caption datasets. We propose an end-to-end framework for describing scene changes from various input modalities, namely, RGB images, depth images, and point cloud data, which are available in most robot applications. We conducted experiments with various input modalities and models and evaluated model performance using datasets with various levels of complexity. Experimental results show that the models that combine RGB images and point cloud data as input achieve high performance in sentence generation and caption correctness and are robust for change type understanding for datasets with high complexity. The developed datasets and models contribute to the study of indoor scene change understanding.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Indoor Scene Change Captioning Based on Multimodality Data

Qiu

Satoh

Suzuki

et al. 2020

Sensors

Self Cite

View full text Add to dashboard Cite

show abstract

“…In comparison, the data size of change detection is more than 100 times smaller than that of the remote sensing image-classification dataset. Second, the image pairs or image sequences used for change detection are often obtained from different viewpoints [26][27][28][29]. In other words, it is difficult to capture a scene from similar viewpoints every time in remote sensing change detection.…”

Section: Introductionmentioning

confidence: 99%

Change Capsule Network for Optical Remote Sensing Image Change Detection

Chen

Zhou

et al. 2021

Remote Sensing

View full text Add to dashboard Cite

Change detection based on deep learning has made great progress recently, but there are still some challenges, such as the small data size in open-labeled datasets, the different viewpoints in image pairs, and the poor similarity measures in feature pairs. To alleviate these problems, this paper presents a novel change capsule network by taking advantage of a capsule network that can better deal with the different viewpoints and can achieve satisfactory performance with small training data for optical remote sensing image change detection. First, two identical non-shared weight capsule networks are designed to extract the vector-based features of image pairs. Second, the unchanged region reconstruction module is adopted to keep the feature space of the unchanged region more consistent. Third, vector cosine and vector difference are utilized to compare the vector-based features in a capsule network efficiently, which can enlarge the separability between the changed pixels and the unchanged pixels. Finally, a binary change map can be produced by analyzing both the vector cosine and vector difference. From the unchanged region reconstruction module and the vector cosine and vector difference module, the extracted feature pairs in a change capsule network are more comparable and separable. Moreover, to test the effectiveness of the proposed change capsule network in dealing with the different viewpoints in multi-temporal images, we collect a new change detection dataset from a taken-over Al Udeid Air Basee (AUAB) using Google Earth. The results of the experiments carried out on the AUAB dataset show that a change capsule network can better deal with the different viewpoints and can improve the comparability and separability of feature pairs. Furthermore, a comparison of the experimental results carried out on the AUAB dataset and SZTAKI AirChange Benchmark Set demonstrates the effectiveness and superiority of the proposed method.

show abstract

Bidirectional difference locating and semantic consistency reasoning for change captioning

Sun

Yao

et al. 2022

Int J of Intelligent Sys

View full text Add to dashboard Cite

Change captioning is an emerging task to describe the changes between a pair of images. The difficulty in this task is to discover the differences between the two images. Recently, some methods have been proposed to address this problem. However, they all employ unidirectional difference localization to identify the changes. This can lead to ambiguity about the nature of the changes. Instead, we propose a framework with bidirectional difference localization and semantic consistency reasoning to describe the image changes. First, we locate the changes in the two images by capturing bidirectional differences. Then we design a decoder with spatial‐channel attention to generate the change caption. Finally, we introduce semantic consistency reasoning to constrain our bidirectional difference localization module and spatial‐channel attention module. Extensive experiments on three public data sets show that the performance of our proposed model outperforms the state‐of‐the‐art change captioning models by a large margin.

show abstract

3D-Aware Scene Change Captioning From Multiview Images

Cited by 23 publications

References 23 publications

Indoor Scene Change Captioning Based on Multimodality Data

Indoor Scene Change Captioning Based on Multimodality Data

Change Capsule Network for Optical Remote Sensing Image Change Detection

Bidirectional difference locating and semantic consistency reasoning for change captioning

Contact Info

Product

Resources

About