AttentionFGAN: Infrared and Visible Image Fusion Using Attention-Based Generative Adversarial Networks

Li, Jing; Huo, Hongtao; Li, Chang; Wang, Ren-Hua; Feng, Qi

doi:10.1109/tmm.2020.2997127

Cited by 214 publications

(67 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In order to settle these issues, Li et al [35] employed a multi-grained attention network with two independent encoders, namely MgAN-Fuse, which integrated a channel attention model into multi-scale layers of the encoder, and then multi-grained attention maps were reconstructed a fused image by the decoder. Subsequently, they extended the attention mechanism into generator and discriminator, termed as At-tentionFGAN [36], which designed two multi-scale attention networks to generate the respective attention maps of infrared and visible images, and were directly concatenated with source images for the fusion network to produce a fused result. These methods only adopt channel attention mechanism to enhance feature representation, but ignore its spatial characteristics.…”

Section: B Gan-based Fusion Methodsmentioning

confidence: 99%

Infrared and Visible Image Fusion via Interactive Compensatory Attention Adversarial Learning

Wang¹,

Shao²,

Chen³

et al. 2022

Preprint

View full text Add to dashboard Cite

The existing generative adversarial fusion methods generally concatenate source images and extract local features through convolution operation, without considering their global characteristics, which tends to produce an unbalanced result and is biased towards the infrared image or visible image. Toward this end, we propose a novel end-to-end mode based on generative adversarial training to achieve better fusion balance, termed as interactive compensatory attention fusion network (ICAFusion). In particular, in the generator, we construct a multi-level encoderdecoder network with a triple path, and adopt infrared and visible paths to provide additional intensity and gradient information. Moreover, we develop interactive and compensatory attention modules to communicate their pathwise information, and model their long-range dependencies to generate attention maps, which can more focus on infrared target perception and visible detail characterization, and further increase the representation power for feature extraction and feature reconstruction. In addition, dual discriminators are designed to identify the similar distribution between fused result and source images, and the generator is optimized to produce a more balanced result. Extensive experiments illustrate that our ICAFusion obtains superior fusion performance and better generalization ability, which precedes other advanced methods in the subjective visual description and objective metric evaluation. Our codes will be public at https://github.com/Zhishe-Wang/ICAFusion.

show abstract

Section: B Gan-based Fusion Methodsmentioning

confidence: 99%

Infrared and Visible Image Fusion via Interactive Compensatory Attention Adversarial Learning

Wang¹,

Shao²,

Chen³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Attention is a method which tends to dispose the most instructive parts of signals among all the accessible computational resources. It has been widely-used in numerous scenes, such as action recognition [20]- [22], visual question answering [23]- [25], and adversarial learning [26]- [29]. Attention grasps long-range contextual information which can be generally applied in different tasks, including machine translation [30], image captioning [31], scene segmentation [32] and object recognition [33].…”

Section: B Attention Mechanismmentioning

confidence: 99%

Over-sampling De-occlusion Attention Network for Prohibited Items Detection in Noisy X-ray Images

Tao,

Wei,

et al. 2021

Preprint

View full text Add to dashboard Cite

Security inspection is X-ray scanning for personal belongings in suitcases, which is significantly important for the public security but highly time-consuming for human inspectors. Fortunately, deep learning has greatly promoted the development of computer vision, offering a possible way of automatic security inspection. However, items within a luggage are randomly overlapped resulting in noisy X-ray images with heavy occlusions. Thus, traditional CNN-based models trained through common image recognition datasets fail to achieve satisfactory performance in this scenario.To address these problems, we contribute the first high-quality prohibited X-ray object detection dataset named OPIXray, which contains 8885 X-ray images from 5 categories of the widely-occurred prohibited item "cutters". The images are gathered from an airport and these prohibited items are annotated manually by professional inspectors, which can be used as a benchmark for model training and further facilitate future research. To better improve occluded X-ray object detection, we further propose an over-sampling deocclusion attention network (DOAM-O), which consists of a novel de-occlusion attention module and a new over-sampling training strategy. Specifically, our de-occlusion module, namely DOAM, simultaneously leverages the different appearance information of the prohibited items; the over-sampling training strategy forces the model to put more emphasis on these hard samples consisting these items of high occlusion levels, which is more suitable for this scenario. We comprehensively evaluated DOAM-O on the OPIXray dataset, which proves that our model can stably improve the performance of the famous detection models such as SSD, YOLOv3, and FCOS, and outperform many extensivelyused attention mechanisms. 1

show abstract

“…The second type as Fig. 14 (b) has one generator and two discriminators such as [116] [117] [118]. Their inputs are the concatenated infrared and visible Images, while one discriminator compares fusion image with visible image and the other compares fusion image with infrared image.…”

Section: Infrared-visible Image Fusionmentioning

confidence: 99%

“…(a) is the architecture with one generator and one discriminator as[114][111][115]. (b) is the architecture with one generator and two discriminators as[116][117][118]. (c) is the coupled GAN architecture as[119][120].…”

mentioning

confidence: 99%

A Review on Remote Sensing Data Fusion with Generative Adversarial Networks (GAN)

Liu¹

2021

Preprint

View full text Add to dashboard Cite

In the past decades, remote sensing (RS) data fusion has always been an active research community. A large number of algorithms and models have been developed. Generative Adversarial Networks (GAN), as an important branch of deep learning, show promising performances in variety of RS image fusions. This review provides an introduction to GAN for remote sensing data fusion. We briefly review the frequently-used architecture and characteristics of GAN in data fusion and comprehensively discuss how to use GAN to realize fusion for homogeneous RS data, heterogeneous RS data, and RS and ground observation data. We also analyzed some typical applications with GAN-based RS image fusion. This review takes insight into how to make GAN adapt to different types of fusion tasks and summarizes the advantages and disadvantages of GAN-based RS data fusion. Finally, we discuss the promising future research directions and make a prediction on its trends.

show abstract

AttentionFGAN: Infrared and Visible Image Fusion Using Attention-Based Generative Adversarial Networks

Cited by 214 publications

References 36 publications

Infrared and Visible Image Fusion via Interactive Compensatory Attention Adversarial Learning

Infrared and Visible Image Fusion via Interactive Compensatory Attention Adversarial Learning

Over-sampling De-occlusion Attention Network for Prohibited Items Detection in Noisy X-ray Images

A Review on Remote Sensing Data Fusion with Generative Adversarial Networks (GAN)

Contact Info

Product

Resources

About