IFCNN: A general image fusion framework based on convolutional neural network

In this paper we propose a novel method for infrared and visible image fusion where we develop nest connectionbased network and spatial/channel attention models. The nest connection-based network can preserve significant amounts of information from input data in a multi-scale perspective. The approach comprises three key elements: encoder, fusion strategy and decoder respectively. In our proposed fusion strategy, spatial attention models and channel attention models are developed that describe the importance of each spatial position and of each channel with deep features. Firstly, the source images are fed into the encoder to extract multi-scale deep features. The novel fusion strategy is then developed to fuse these features for each scale. Finally, the fused image is reconstructed by the nest connection-based decoder. Experiments are performed on publicly available datasets. These exhibit that our proposed approach has better fusion performance than other state-of-theart methods. This claim is justified through both subjective and objective evaluation. The code of our fusion method is available at https://github.com/hli1221/imagefusion-nestfuse.

show abstract

“…(3) The end-to-end image fusion frameworks [26] [27] can generate the fused image without any handcrafted feature extraction operation.…”

Section: With Training Phasementioning

confidence: 99%

NestFuse: An Infrared and Visible Image Fusion Architecture Based on Nest Connection and Spatial/Channel Attention Models

Durrani

2020

IEEE Trans. Instrum. Meas.

416

160

View full text Add to dashboard Cite

show abstract

“…SAE-based DL models have been applied to image fusion for multimodal medical image feature extraction. These extracted features can be used to design optimal fusion rules [159] and to obtain better fusion images. In this method, a multitask loss function related to image fusion quality is used to train the network.…”

Section: Stacked Autoencodersmentioning

confidence: 99%

Application of Image Fusion in Diagnosis and Treatment of Liver Cancer

Zhu

2020

Applied Sciences

View full text Add to dashboard Cite

With the accelerated development of medical imaging equipment and techniques, image fusion technology has been effectively applied for diagnosis, biopsy and radiofrequency ablation, especially for liver tumor. Tumor treatment relying on a single medical imaging modality might face challenges, due to the deep positioning of the lesions, operation history and the specific background conditions of the liver disease. Image fusion technology has been employed to address these challenges. Using the image fusion technology, one could obtain real-time anatomical imaging superimposed by functional images showing the same plane to facilitate the diagnosis and treatments of liver tumors. This paper presents a review of the key principles of image fusion technology, its application in tumor treatments, particularly in liver tumors, and concludes with a discussion of the limitations and prospects of the image fusion technology.

show abstract

“…However, in the task of fusing PET and MRI images, it is nearly impossible to establish the ground truth due to defining a standard for final fused images is unrealistic. Moreover, in order to complete the image fusion task, most of the proposed CNN models require additional post-processing procedures because they are not designed in the end-to-end manner [24].…”

Section: Introductionmentioning

confidence: 99%

Fusion of Brain PET and MRI Images Using Tissue-Aware Conditional Generative Adversarial Network With Joint Loss

Kang¹,

Lü

Zhang

2020

IEEE Access

View full text Add to dashboard Cite

Positron emission tomography (PET) has rich pseudo color information that reflects the functional characteristics of tissue, but lacks structural information and its spatial resolution is low. Magnetic resonance imaging (MRI) has high spatial resolution as well as strong structural information of soft tissue, but lacks color information that shows the functional characteristics of tissue. For the purpose of integrating the color information of PET with the anatomical structures of MRI to help doctors diagnose diseases better, a method for fusing brain PET and MRI images using tissue-aware conditional generative adversarial network (TA-cGAN) is proposed. Specifically, the process of fusing brain PET and MRI images is treated as an adversarial machine between retaining the color information of PET and preserving the anatomical information of MRI. More specifically, the fusion of PET and MRI images can be regarded as a min-max optimization problem with respect to the generator and the discriminator, where the generator attempts to minimize the objective function via generating a fused image mainly contains the color information of PET, whereas the discriminator tries to maximize the objective function through urging the fused image to include more structural information of MRI. Both the generator and the discriminator in TA-cGAN are conditioned on the tissue label map generated from MRI image, and are trained alternatively with joint loss. Extensive experiments demonstrate that the proposed method enhances the anatomical details of the fused image while effectively preserving the color information from the PET. In addition, compared with other state-of-the-art methods, the proposed method achieves better fusion effects both in subjectively visual perception and in objectively quantitative assessment.

show abstract

IFCNN: A general image fusion framework based on convolutional neural network

Cited by 835 publications

References 40 publications

NestFuse: An Infrared and Visible Image Fusion Architecture Based on Nest Connection and Spatial/Channel Attention Models

NestFuse: An Infrared and Visible Image Fusion Architecture Based on Nest Connection and Spatial/Channel Attention Models

Application of Image Fusion in Diagnosis and Treatment of Liver Cancer

Fusion of Brain PET and MRI Images Using Tissue-Aware Conditional Generative Adversarial Network With Joint Loss

Contact Info

Product

Resources

About