Late fusion of deep learning and handcrafted visual features for biomedical image modality classification

Lee, Sheng Long; Zare, Mohammad Reza; Müller, Henning

doi:10.1049/iet-ipr.2018.5054

Cited by 19 publications

(8 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…But both networks have their own advantages, it is difficult to decide which one is more reliable. Late fusion is quite common in biomedical image modality classification [45], we explore more score‐based fusion methods for global and local problems, taking the maximum of the two scores as the final result as well as adding the two scores by weight. The fusion algorithms are defined as

\begin{matrix} right left right left right left right left right left right left0.278em 2em 0.278em 2em 0.278em 2em 0.278em 2em 0.278em 2em 0.278em3pt \\ r e l_s c o r e_{i} & = \max (g_s c o r e_{i}, l_s c o r e_{i}) \end{matrix}

\begin{matrix} right left right left right left right left right left right left0.278em 2em 0.278em 2em 0.278em 2em 0.278em 2em 0.278em 2em 0.278em3pt \\ r e l_s c o r e_{i} & = ω_{r} g_s c o r e_{i} + (1 - ω_{r}) l_s c o r e_{i} \end{matrix}

where

r e l_s c o r e_{i}

is the score result corresponding to the i th label,

g_s c o r e_{i}

and

l_s c o r e_{i}

are the global score and local score corresponding to the i th label,

ω_{r}

is the weight assigned to the global score.…”

Section: Methodsmentioning

confidence: 99%

Visual saliency based global–local feature representation for skin cancer classification

Xiao

2020

IET image process

View full text Add to dashboard Cite

\begin{matrix} right left right left right left right left right left right left0.278em 2em 0.278em 2em 0.278em 2em 0.278em 2em 0.278em 2em 0.278em3pt \\ r e l_s c o r e_{i} & = \max (g_s c o r e_{i}, l_s c o r e_{i}) \end{matrix}

\begin{matrix} right left right left right left right left right left right left0.278em 2em 0.278em 2em 0.278em 2em 0.278em 2em 0.278em 2em 0.278em3pt \\ r e l_s c o r e_{i} & = ω_{r} g_s c o r e_{i} + (1 - ω_{r}) l_s c o r e_{i} \end{matrix}

where

r e l_s c o r e_{i}

is the score result corresponding to the i th label,

g_s c o r e_{i}

and

l_s c o r e_{i}

are the global score and local score corresponding to the i th label,

ω_{r}

is the weight assigned to the global score.…”

Section: Methodsmentioning

confidence: 99%

Visual saliency based global–local feature representation for skin cancer classification

Xiao

2020

IET image process

View full text Add to dashboard Cite

“…The cross-media feature fusion approaches [10][11][12][13][14][15][16][17] can be roughly divided into two groups: early fusion based and late fusion based. The early fusion based approaches combine feature representations of different media data into an unified feature representation before classification, while the late fusion based approaches combine the results of the classifiers for different media data.…”

Section: Cross-media Feature Fusionmentioning

confidence: 99%

A deep grouping fusion neural network for multimedia content understanding

Song

Shang

et al. 2022

IET Image Processing

View full text Add to dashboard Cite

How Deep Neural Networks (DNNs) best cope with the understanding of multimedia contents still remains an open problem, mainly due to two factors. First, conventional DNNs cannot effectively learn the representations of the images with sparse visual information. For example, the images describing knowledge concepts in textbooks. Second, existing DNNs cannot effectively capture the fine‐grained interactions between the images and text descriptions. To address these issues, we propose a deep Cross‐Media Grouping Fusion Network (CMGFN), which mainly has two distinctive properties: 1) CMGFN can effectively learn visual features from the images with sparse visual information. This is achieved by first progressively adjusting the attention of convolution filters to valuable visual regions, and then enhancing the use of key visual information in feature construction. 2) By a cross‐media grouping co‐attention mechanism, CMGFN can effectively use the interactions between visual features of different semantics and textual descriptions, to learn cross‐media features representing different fine‐grained semantics in different groups. Empirical studies demonstrate that CMGFN not only achieves state‐of‐the‐art performance on the multimedia documents containing sparse visual information, but also shows superior general applicability on other multimedia data, e.g., the multimedia fake news.

show abstract

“…Deep learning (DL) approaches can be used as a late step in most fusion strategies (Lee, Mohammad & Henning, 2018). Most of CT and CXR images in medical applications can be handcrafted and fuzed in score level fusion strategy (Baumgartl et al, 2020).…”

Section: Deep Learning In Image Fusion Strategiesmentioning

confidence: 99%

A novel perceptual two layer image fusion using deep learning for imbalanced COVID-19 dataset

Elzeki

Elfattah²,

Salem

et al. 2021

PeerJ Computer Science

View full text Add to dashboard Cite

Background and Purpose COVID-19 is a new strain of viruses that causes life stoppage worldwide. At this time, the new coronavirus COVID-19 is spreading rapidly across the world and poses a threat to people’s health. Experimental medical tests and analysis have shown that the infection of lungs occurs in almost all COVID-19 patients. Although Computed Tomography of the chest is a useful imaging method for diagnosing diseases related to the lung, chest X-ray (CXR) is more widely available, mainly due to its lower price and results. Deep learning (DL), one of the significant popular artificial intelligence techniques, is an effective way to help doctors analyze how a large number of CXR images is crucial to performance. Materials and Methods In this article, we propose a novel perceptual two-layer image fusion using DL to obtain more informative CXR images for a COVID-19 dataset. To assess the proposed algorithm performance, the dataset used for this work includes 87 CXR images acquired from 25 cases, all of which were confirmed with COVID-19. The dataset preprocessing is needed to facilitate the role of convolutional neural networks (CNN). Thus, hybrid decomposition and fusion of Nonsubsampled Contourlet Transform (NSCT) and CNN_VGG19 as feature extractor was used. Results Our experimental results show that imbalanced COVID-19 datasets can be reliably generated by the algorithm established here. Compared to the COVID-19 dataset used, the fuzed images have more features and characteristics. In evaluation performance measures, six metrics are applied, such as QAB/F, QMI, PSNR, SSIM, SF, and STD, to determine the evaluation of various medical image fusion (MIF). In the QMI, PSNR, SSIM, the proposed algorithm NSCT + CNN_VGG19 achieves the greatest and the features characteristics found in the fuzed image is the largest. We can deduce that the proposed fusion algorithm is efficient enough to generate CXR COVID-19 images that are more useful for the examiner to explore patient status. Conclusions A novel image fusion algorithm using DL for an imbalanced COVID-19 dataset is the crucial contribution of this work. Extensive results of the experiment display that the proposed algorithm NSCT + CNN_VGG19 outperforms competitive image fusion algorithms.

show abstract

Late fusion of deep learning and handcrafted visual features for biomedical image modality classification

Cited by 19 publications

References 30 publications

Visual saliency based global–local feature representation for skin cancer classification

Visual saliency based global–local feature representation for skin cancer classification

A deep grouping fusion neural network for multimedia content understanding

A novel perceptual two layer image fusion using deep learning for imbalanced COVID-19 dataset

Contact Info

Product

Resources

About