SuperFusion: A Versatile Image Registration and Fusion Network with Semantic Awareness

Tang, Linfeng; Deng, Yuxin; Ma, Yong; Huang, Jun; Ma, Jiayi

doi:10.1109/jas.2022.106082

Cited by 152 publications

(19 citation statements)

References 68 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Due to the excellent feature learning ability and nonlinear fitting ability of neural networks, researchers have explored data-driven infrared and visible image fusion methods based on deep learning [1], [49], [50]. These methods mainly include AE-based methods [18], [51], [52], CNN-based methods [53], [27], [4], and GAN-based methods [9], [54], [20].…”

Section: A Infrared and Visible Image Fusionmentioning

confidence: 99%

Dif-Fusion: Towards High Color Fidelity in Infrared and Visible Image Fusion with Diffusion Models

Yue¹,

Fang²,

Xia³

et al. 2023

Preprint

View full text Add to dashboard Cite

Color plays an important role in human visual perception, reflecting the spectrum of objects. However, the existing infrared and visible image fusion methods rarely explore how to handle multi-spectral/channel data directly and achieve high color fidelity. This paper addresses the above issue by proposing a novel method with diffusion models, termed as Dif-Fusion, to generate the distribution of the multi-channel input data, which increases the ability of multi-source information aggregation and the fidelity of colors. In specific, instead of converting multi-channel images into single-channel data in existing fusion methods, we create the multi-channel data distribution with a denoising network in a latent space with forward and reverse diffusion process. Then, we use the denoising network to extract the multi-channel diffusion features with both visible and infrared information. Finally, we feed the multi-channel diffusion features to the multi-channel fusion module to directly generate the three-channel fused image. To retain the texture and intensity information, we propose multi-channel gradient loss and intensity loss. Along with the current evaluation metrics for measuring texture and intensity fidelity, we introduce a new evaluation metric to quantify color fidelity. Extensive experiments indicate that our method is more effective than other state-of-the-art image fusion methods, especially in color fidelity. The source code will be publicly available at https://github.com/GeoVectorMatrix/Dif-Fusion.

show abstract

Section: A Infrared and Visible Image Fusionmentioning

confidence: 99%

Dif-Fusion: Towards High Color Fidelity in Infrared and Visible Image Fusion with Diffusion Models

Yue¹,

Fang²,

Xia³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…In order to improve the generalization ability of the training model, 3,000 image pairs are randomly selected from the two datasets respectively, and a total of 6,000 image pairs from the training set of the proposed algorithm in this work. To verify the effectiveness of the method, 49 pairs of widely used infrared and visible images are randomly selected from the three datasets TNO 3 , VOT2020-RGBT 4 and RoadScence 5 to construct the test set in this work. Among them, 39 pairs of images are from TNO and VOT2020-RGBT datasets and 10 pairs of images are from the RoadScence dataset.…”

Section: Datasetmentioning

confidence: 99%

“…In recent years, with the rapid development of deep learning, the research of fusion methods among diverse modal information has made significant progress [4][5][6][7][8]. As an important branch in the field of image fusion, infrared and visible image fusion has attracted the attention of researchers, and a series of effective methods have been proposed.…”

Section: Introductionmentioning

confidence: 99%

Infrared and visible image fusion with edge detail implantation

Liu

Zhang

2023

Front. Phys.

View full text Add to dashboard Cite

Infrared and visible image fusion aims to integrate complementary information from the same scene images captured by different types of sensors into one image to obtain a fusion image with richer information. Recently, deep learning-based infrared and visible image fusion methods have been widely used. However, it is still a difficult problem how to maintain the edge detail information in the source images more effectively. To address this problem, we propose a novel infrared and visible image fusion method with edge detail implantation. The proposed method no longer improves the performance of edge details in the fused image through making the extracted features contain edge detail information like traditional methods, but by processing source image information and edge detail information separately, and supplementing edge details to the main framework. Technically, we propose a two-branch feature representation framework. One branch is used to directly extract features from the input source image, while the other is utilized to extract features of edge map. The edge detail branch mainly provides edge detail features for the source image input branch, ensuring that the output features contain rich edge detail information. In the fusion of multi-source features, we respectively fuse the source image features and the edge detail features, and use the fusion results of edge details to guide and enhance the fusion results of source image features so that they contain richer edge detail information. A large number of experimental results demonstrate the effectiveness of the proposed method.

show abstract

“…The interaction between neurons represents the transmission and processing of neuron information, thus making the neural network highly adaptable, with fault tolerance and anti-noise capabilities [19], [20], [21]. The DL-based method has been substantially improved and achieved good research results in the past five years [3], [4], [9], [22], [23], [24], [25], [26], [27], [28], [29], [30]. Liu et al [24] applied the deep convolutional neural network (CNN) to the IVIF for the first time, which can deal with activity level measurement and weight assignment to overcome the difficulty of manual design.…”

Section: Introductionmentioning

confidence: 99%

“…Ma et al [27] proposed an end-to-end model, STDFusionNet, that first calculate salient target mask and then design a specific loss function to guide the feature extraction and reconstruction. Tang et al [28] proposed a SuperFusion that registration network and a Lovasz-Softmax loss are combined used to implement a practical network to meet registration, fusion, and semantic requirements. Several transformer-based fusion algorithms [29], [30], in recently years, have been developed.…”

Section: Introductionmentioning

confidence: 99%

A Target-Aware Fusion Framework for Infrared and Visible Images

Zhang

Lee

2023

IEEE Access

View full text Add to dashboard Cite

Infrared and visible image fusion aims to obtain an image that can retain the prominent infrared target and the detailed texture information from the source images. Most scale filter-based decomposition methods mainly attempt to extract more detailed features by increasing decomposition layers, and they ignore fully considering the intrinsic properties of the original images and the influence of noises. To solve this issue and obtain better fusion & detection performance, this paper proposes a fusion method via a globalet filter and detail enhancement model to construct a target-aware fusion framework. The globalet filter is first comprehensively considered from three perspectives based on global point-to-point optimization: target brightness, removal of gradients of large-small scales (noise reduction), and preservation of texture details. Mathematically, three limited error measure equations are constructed between the target output and the source image in the shape of the L 2 -norm and the first-order derivative difference. Next, a weighted average operator and a detail enhancement model are proposed to guide the corresponding sub-layers. This model creates connections between the detail layers and the input images so that the fused detail layer contains the best pixel region from these images to the greatest extent possible after the ''maximum absolute'' rule. As a result, the fused image is reconstructed by adding the previously obtained sub-images. Extensive experimental results demonstrate that our method outperforms state-of-the-art fusion methods, particularly in highlighting infrared targets, preserving substantial details, and producing average detection accuracy of exceeds 98%.INDEX TERMS Infrared and visible image fusion, target-aware fusion framework, a globalet filter, detail enhancement model.

show abstract

SuperFusion: A Versatile Image Registration and Fusion Network with Semantic Awareness

Cited by 152 publications

References 68 publications

Dif-Fusion: Towards High Color Fidelity in Infrared and Visible Image Fusion with Diffusion Models

Dif-Fusion: Towards High Color Fidelity in Infrared and Visible Image Fusion with Diffusion Models

Infrared and visible image fusion with edge detail implantation

A Target-Aware Fusion Framework for Infrared and Visible Images

Contact Info

Product

Resources

About