Multimodal medical image fusion review: Theoretical background and recent advances

Hermessi, Haithem; Mourali, Olfa; Zagrouba, Ezzeddine

doi:10.1016/j.sigpro.2021.108036

Cited by 158 publications

(62 citation statements)

References 134 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Besides, as for our MMIDFNet method, the accuracy of the data fusion model (0.846) is 3.2% lower than that of the decision fusion model (0.878) via 3-fold cross-validation. This is mainly caused by the limitations of the data-level fusion strategy that does not fully take advantage of the features underlying each modality data and does not deal with how to fuse the features from the multimodal MRI images ( 33 , 43 , 44 ). However, in our decision fusion model, we used the weighting manner to ensemble the unimodal models in inferring stage.…”

Section: Discussionmentioning

confidence: 99%

Multimodal MRI Image Decision Fusion-Based Network for Glioma Classification

et al. 2022

View full text Add to dashboard Cite

PurposeGlioma is the most common primary brain tumor, with varying degrees of aggressiveness and prognosis. Accurate glioma classification is very important for treatment planning and prognosis prediction. The main purpose of this study is to design a novel effective algorithm for further improving the performance of glioma subtype classification using multimodal MRI images.MethodMRI images of four modalities for 221 glioma patients were collected from Computational Precision Medicine: Radiology-Pathology 2020 challenge, including T1, T2, T1ce, and fluid-attenuated inversion recovery (FLAIR) MRI images, to classify astrocytoma, oligodendroglioma, and glioblastoma. We proposed a multimodal MRI image decision fusion-based network for improving the glioma classification accuracy. First, the MRI images of each modality were input into a pre-trained tumor segmentation model to delineate the regions of tumor lesions. Then, the whole tumor regions were centrally clipped from original MRI images followed by max–min normalization. Subsequently, a deep learning-based network was designed based on a unified DenseNet structure, which extracts features through a series of dense blocks. After that, two fully connected layers were used to map the features into three glioma subtypes. During the training stage, we used the images of each modality after tumor segmentation to train the network to obtain its best accuracy on our testing set. During the inferring stage, a linear weighted module based on a decision fusion strategy was applied to assemble the predicted probabilities of the pre-trained models obtained in the training stage. Finally, the performance of our method was evaluated in terms of accuracy, area under the curve (AUC), sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), etc.ResultsThe proposed method achieved an accuracy of 0.878, an AUC of 0.902, a sensitivity of 0.772, a specificity of 0.930, a PPV of 0.862, an NPV of 0.949, and a Cohen’s Kappa of 0.773, which showed a significantly higher performance than existing state-of-the-art methods.ConclusionCompared with current studies, this study demonstrated the effectiveness and superiority in the overall performance of our proposed multimodal MRI image decision fusion-based network method for glioma subtype classification, which would be of enormous potential value in clinical practice.

show abstract

Section: Discussionmentioning

confidence: 99%

Multimodal MRI Image Decision Fusion-Based Network for Glioma Classification

et al. 2022

View full text Add to dashboard Cite

show abstract

“…feature maps F 0 and F 1 are multiplied by CAM probabilities P c0,I0 and P c1,I1 respectively). FM3, Fusion Method 3: Fusion using CAMs utilising (7),( 8),( 9), (10) and (11).…”

Section: Methods Definitionsmentioning

confidence: 99%

“…More recently, state-of-the-art image fusion techniques have focused on the use of network based methods [10,11,12,13,14,15,16,17]. Due to the requirements of needing training data these methods are often domain focused with state-of-the-art results reported for IR/Visible fusion [12,13], remote sensing (multispectral) [14] and multi-focus areas [15,16].…”

Section: Introductionmentioning

confidence: 99%

Semantic Image Fusion

Hill,

Bull

2021

Preprint

View full text Add to dashboard Cite

Image fusion methods and metrics for their evaluation have conventionally used pixel based or low level features. However, for many applications the aim of image fusion is to effectively combine the semantic content of the input images. This paper proposes a novel system for the semantic combination of visual content using pre-trained CNN network architectures. Our proposed semantic fusion is initiated through the fusion of the top layer feature map outputs (for each input image) through gradient updating of the fused image input (so called image optimisation). Simple 'choose maximum' and 'local majority' filter based fusion rules are utilised for feature map fusion. This provides a simple method to combine layer outputs and thus a unique framework to fuse single channel and colour images within a decomposition pre-trained for classification and therefore aligned with semantic fusion. Furthermore, class activation mappings of each input image are used to combine semantic information at a higher level. The developed methods are able to give equivalent low level fusion performance to state of the art methods while providing a unique architecture to combine semantic information from multiple images.

show abstract

“…According to the different stages of fusion processing, fusion algorithms can be divided into three categories: fusion based on pixel level, fusion based on feature level, and fusion based on decision level [8]. Pixel-level fusion is performed directly on the original data layer.…”

Section: Introductionmentioning

confidence: 99%

FSADFuse: A Novel Fusion Approach to Infrared and Visible Images

Hao

Tian

et al. 2021

IEEE Access

View full text Add to dashboard Cite

To address the problems of edge blur and weak detail resolution when fusing infrared and visible images with traditional methods, a novel image fusion approach based on fast super-resolution convolutional neural network and anisotropic diffusion is proposed. The proposed method first employs super-resolution reconstruction based on fast convolutional neural networks to get high-resolution infrared images. Second, the visible and high-resolution infrared images are anisotropically diffused to obtain the corresponding base layers and detail layers. Next, the fusion of the detail layer is performed through Karhunen-Loeve transform, and a saliency adaptive module is constructed to fuse the base layer. Finally, the fusion result is obtained by applying linear reconstruction on the fused detail layer and base layer. Compared with other algorithms, the experimental results show that the proposed algorithm not only has better objective evaluation indexes, but also can better retain the details of the original image. INDEX TERMS Image fusion, infrared and visible images, convolution neural networks, super-resolution reconstruction, multi-scale decomposition.

show abstract

Multimodal medical image fusion review: Theoretical background and recent advances

Cited by 158 publications

References 134 publications

Multimodal MRI Image Decision Fusion-Based Network for Glioma Classification

Multimodal MRI Image Decision Fusion-Based Network for Glioma Classification

Semantic Image Fusion

FSADFuse: A Novel Fusion Approach to Infrared and Visible Images

Contact Info

Product

Resources

About