A Multi-Stage Visible and Infrared Image Fusion Network Based on Attention Mechanism

Abstract

Pixel-level image fusion is an effective way to fully exploit the rich texture information of visible images and the salient target characteristics of infrared images. With the development of deep learning technology in recent years, the image fusion algorithm based on this method has also achieved great success. However, owing to the lack of sufficient and reliable paired data and a nonexistent ideal fusion result as supervision, it is difficult to design a precise network training mode. Moreover, the manual fusion strategy has difficulty ensuring the full use of information, which easily causes redundancy and omittance. To solve the above problems, this paper proposes a multi-stage visible and infrared image fusion network based on an attention mechanism (MSFAM). Our method stabilizes the training process through multi-stage training and enhances features by the learning attention fusion block. To improve the network effect, we further design a Semantic Constraint module and Push–Pull loss function for the fusion task. Compared with several recently used methods, the qualitative comparison intuitively shows more beautiful and natural fusion results by our model with a stronger applicability. For quantitative experiments, MSFAM achieves the best results in three of the six frequently used metrics in fusion tasks, while other methods only obtain good scores on a single metric or a few metrics. Besides, a commonly used high-level semantic task, i.e., object detection, is used to prove its greater benefits for downstream tasks compared with singlelight images and fusion results by existing methods. All these experiments prove the superiority and effectiveness of our algorithm.

A multimodal fusion-based autoencoder for nondestructive evaluation of aircraft structures

“…Another study proposes a fusion algorithm based on the adaptive dual-channel unit-linking pulse coupled neural network (PCNN) for infrared and visible images fusion in nonsubsampled contourlet transform (NSCT) domain [13]. A multi-stage visible and infrared image fusion network based on an attention mechanism (MSFAM) to refine fusion features is shown in [14]. However, the methods for NDE fusion are few and the paucity has promoted investigations into some of the most advanced infrared and visual image fusion techniques.…”

Image fusion using Y-net-based extractor and global-local discriminator

“…Recently, it is a tendency to build performance-efficient deep neural networks for various image fusion tasks due to their strong nonlinear learning abilities. Learning-based fusion architectures, such as autoencoder (AE) [ 13 , 14 , 16 , 19 ], convolutional neural network (CNN) [ 15 , 18 , 20 ] and generative adversarial network (GAN) [ 21 , 22 , 24 , 27 , 29 ] have witnessed obvious improvements in fusion performance, but their single-scale frameworks can hardly capture the full-scale features of the real-world targets and fail to make the fused images photorealistic. More importantly, most methods directly capitalize on the features extracted in the last layer to reconstruct fused images, whereas earlier features do not.…”

“…U-Net is originally proposed for image segmentation tasks [ 31 ]. With the powerful multi-scale representation advantage, more and more computer vision tasks are realized by using U-Net as the backbone network, such as image dehazing [ 32 ], salient object detection [ 33 ], facial emotion recognition [ 34 ], image denoising [ 35 ], image fusion [ 13 , [36] , [37] , [38] , [39] ]. U-Net architecture adopts a symmetric encoder-decoder manner that overcomes the disadvantages of local and global features loss in fully convolutional networks.…”

“…Fu [ 12 ] extracted the different level features via d dense connection operations. Zheng [ 13 ] achieved feature extraction at different scales and levels using HINBlock. Reference [ [14] , [15] , [16] , [17] ] employed convolution kernels of different sizes to extract common and unique features of source images.…”

DSA-Net: Infrared and Visible Image Fusion via Dual-Stream Asymmetric Network

“…Therefore, extracting the significant features of the fusion image is one of the central problems. Over the past few decades, numerous fusion methods have been proposed by researchers, which can be roughly divided into two categories: traditional fusion methods [ 5 , 6 , 7 ] and deep learning-based fusion methods [ 8 , 9 , 10 ]. Traditional fusion methods measure pixels’ salience in the spatial domain or transform domain, and later design specific fusion rules to fuse them to obtain the fused image.…”