Multimodal image fusion via coupled feature learning

Veshki, Farshad G.; Ouzir, Nora; Vorobyov, Sergiy A.; Ollila, Esa

doi:10.1016/j.sigpro.2022.108637

Cited by 31 publications

(13 citation statements)

References 44 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this section, the performance of the proposed method ICIF is verified with qualitative and quantitative evaluations. The compared methods include MDLatLRR [17], ResNetFusion [35], GANMcC [35], NestFuse [39], SEDRFuse [37], STDFusionNet [36], FusionGAN [45], RTVD-VIF [57], and the MMIF [58]. To ensure an objective evaluation, the following evaluation indexes are used, namely, average gradient (AG) [59], information entropy (H) [60], standard deviation (SD) [61], spatial frequency (SF) [62], edge strength (EI) [63], fusion loss function (L AB/F ) [56], fusion volume function (Q AB/F ) [56], and the artifact function (N AB/F ) [56].…”

Section: Resultsmentioning

confidence: 99%

ICIF: Image fusion via information clustering and image features

et al. 2023

View full text Add to dashboard Cite

Image fusion technology is employed to integrate images collected by utilizing different types of sensors into the same image to generate high-definition images and extract more comprehensive information. However, all available techniques derive the features of the images by utilizing each sensor separately, resulting in poorly correlated image features when different types of sensors are utilized during the fusion process. The fusion strategy to make up for the differences between features alone is an important reason for the poor clarity of fusion results. Therefore, this paper proposes a fusion method via information clustering and image features (ICIF). First, the weighted median filter algorithm is adopted in the spatial domain to realize the clustering of images, which uses the texture features of an infrared image as the weight to influence the clustering results of the visible light image. Then, the image is decomposed into the base layer, bright detail layer, and dark detail layer, which improves the correlations between the layers after conducting the decomposition of a source graph. Finally, the characteristics of the images collected by utilizing sensors and feature information between the image layers are used as the weight reference of the fusion strategy. Hence, the fusion images are reconstructed according to the principle of extended texture details. Experiments on public datasets demonstrate the superiority of the proposed strategy over state-of-the-art methods. The proposed ICIF highlighted targets and abundant details as well. Moreover, we also generalize the proposed ICIF to fuse images with different sensors, e.g., medical images and multi-focus images.

show abstract

Section: Resultsmentioning

confidence: 99%

ICIF: Image fusion via information clustering and image features

et al. 2023

View full text Add to dashboard Cite

show abstract

“…learning based infrared and visible image fusion method (MMIF) [29]. This method decomposes the source image into correlated and uncorrelated components, and does not require any training data.…”

Section: Veshki Et Al Proposed a Coupled Dictionarymentioning

confidence: 99%

“…In general, visible frame contains more detail than infrared frame, however, under low light conditions, infrared frame will contain more details, so Equation ( 28) needs to be improved. In this paper, the standard deviation [49] is used to measure how much detail information is contained in the two kinds of frames, and the standard deviation is calculated as shown in Equation (29).…”

Section: Controller Designmentioning

confidence: 99%

Infrared and visible video fusion method based on local differential correlation features and PID control

Tang,

Wang

2024

Preprint

View full text Add to dashboard Cite

The purpose of infrared and visible video fusion is to combine the complementary features of videos from different modalities. Most fusion algorithms ignore the feature associations of adjacent frames and the guidance of the source frames to the fusion process. Therefore, this paper proposes a new fusion method. First, a spatial-domain multi-attribute detail feature extraction model is proposed, which is capable of extracting the base layer, the bright detail layer and the dark detail layer of the source frames separately. Then, we propose an associated feature extraction model for adjacent frames, which improves the spatial continuity of the fused video. Furthermore, an exponential homomorphic filter is proposed, which simultaneously increases the dynamic range and contrast of the source infrared frames to obtain a general salient target extraction model. In the feature fusion stage, a weighted fusion rule based on edge intensity is used in the detail layer. Then we design the controller, transfer function and measurement function separately, so as to construct a closed-loop proportional-integral-derivative (PID) control system to fuse the base layer, which ensures that the fused video maintains more information of the source video. Experiments on public datasets demonstrate that our fusion method outperforms some state-of-the-art algorithms.

show abstract

“…Similarly, Liu et al [34] introduced channel and spatial dual attention mechanisms to fuse 3D multimodal information, and designed specific loss functions for different modality features. Veshki et al [35] divided the images into relevant and irrelevant components. For the relevant components, the maximum absolute value rule was adopted.…”

Section: Multimodal Feature Fusionmentioning

confidence: 99%

RFTNet: Region–Attention Fusion Network Combined with Dual-Branch Vision Transformer for Multimodal Brain Tumor Image Segmentation

Jiao,

Yang,

Yan

et al. 2023

Electronics

View full text Add to dashboard Cite

Brain tumor image segmentation plays a significant auxiliary role in clinical diagnosis. Recently, deep learning has been introduced into multimodal segmentation tasks, which construct various Convolutional Neural Network (CNN) structures to achieve excellent performance. However, most CNN-based segmentation methods have poor capability for global feature extraction. Transformer is good at modeling long-distance dependencies, but it can cause local information loss and usually has a high computational complexity. In addition, it is difficult to fully exploit the brain tumor features of different modalities. To address these issues, in this paper, we propose a region–attention fusion (RAF) network that combines a dual-branch vision Transformer (DVT), called RFTNet. In RFTNet, the DVT is exploited to capture the delicate local information and global semantics separately by two branches. Meanwhile, a novel RAF is employed to effectively fuse the images of the different modalities. Finally, we design a new hybrid loss function, called region-mixed loss function (RML) to calculate the importance of each pixel and solve the problem of class imbalance. The experiments on BrasTS2018 and BraTS2020 datasets show that our method obtains a higher segmentation accuracy than other models. Furthermore, ablation experiments prove the effectiveness of each key component in RFTNet.

show abstract

Multimodal image fusion via coupled feature learning

Cited by 31 publications

References 44 publications

ICIF: Image fusion via information clustering and image features

ICIF: Image fusion via information clustering and image features

Infrared and visible video fusion method based on local differential correlation features and PID control

RFTNet: Region–Attention Fusion Network Combined with Dual-Branch Vision Transformer for Multimodal Brain Tumor Image Segmentation

Contact Info

Product

Resources

About