Hierarchical Progressive Network for Multimodal Medical Image Fusion in Healthcare Systems

Yang, Sihan; Yang, Xiaobo; Zhang, Rongzhu; Liu, Kai

doi:10.1109/tcss.2022.3165559

Cited by 5 publications

(3 citation statements)

References 62 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Chakraborty and Kishor [ 18 ] demonstrated a cloud-fog-based IoMT system for cardiac disease prediction using machine learning classification techniques. To combine many medical images into one, a fusion network based on deep learning is proposed [ 19 ]. Since the suggested fusion architecture is unsupervised, custom fusion rules are unnecessary.…”

Section: Related Workmentioning

confidence: 99%

FedHealthFog: A federated learning-enabled approach towards healthcare analytics over fog computing platform

Tripathy,

Bebortta,

Chowdhary

et al. 2024

Heliyon

View full text Add to dashboard Cite

Section: Related Workmentioning

confidence: 99%

FedHealthFog: A federated learning-enabled approach towards healthcare analytics over fog computing platform

Tripathy,

Bebortta,

Chowdhary

et al. 2024

Heliyon

View full text Add to dashboard Cite

“…The multimodal fusion strategy significantly affects the performance of sEMG-ACC-based HGR. Early fusion and late fusion are two typical fusion strategies that perform modality fusion in the early and late stages, respectively [19]. However, these strategies are unable to sufficiently characterize intramodal specificity and cross-modal association at the same time [20].…”

Section: Introductionmentioning

confidence: 99%

“…It is widely reported that the correlation of modalities can promote the performance of multimodal methods [21]. The effectiveness of sEMG-ACC-based HGR may be further enhanced when the interactive relationships are deeply explored [19], [22]. In addition, the recognition performance of multimodal HGR is still suffering from poor robustness, especially the problem of missing modality.…”

Section: Introductionmentioning

confidence: 99%

Alignment-Enhanced Interactive Fusion Model for Complete and Incomplete Multimodal Hand Gesture Recognition

Duan,

Wu,

Liu

et al. 2023

IEEE Trans. Neural Syst. Rehabil. Eng.

View full text Add to dashboard Cite

Hand gesture recognition (HGR) based on surface electromyogram (sEMG) and Accelerometer (ACC) signals is increasingly attractive where fusion strategies are crucial for performance and remain challenging. Currently, neural network-based fusion methods have gained superior performance. Nevertheless, these methods typically fuse sEMG and ACC either in the early or late stages, overlooking the integration of entire cross-modal hierarchical information within each individual hidden layer, thus inducing inefficient inter-modal fusion. To this end, we propose a novel Alignment-Enhanced Interactive Fusion (AiFusion) model, which achieves effective fusion via a progressive hierarchical fusion strategy. Notably, AiFusion can flexibly perform both complete and incomplete multimodal HGR. Specifically, AiFusion contains two unimodal branches and a cascaded transformer-based multimodal fusion branch. The fusion branch is first designed to adequately characterize modality-interactive knowledge by adaptively capturing inter-modal similarity and fusing hierarchical features from all branches layer by layer. Then, the modality-interactive knowledge is aligned with that of unimodality using cross-modal supervised contrastive learning and online distillation from embedding and probability spaces respectively. These alignments further promote fusion quality and refine modality-specific representations. Finally, the recognition outcomes are set to be determined by available modalities, thus contributing to handling the incomplete multimodal HGR problem, which is frequently encountered in real-world scenarios. Experimental results on five public datasets demonstrate that AiFusion outperforms most state-of-the-art benchmarks in complete multimodal HGR. Impressively, it also surpasses the unimodal baselines in the challenging incomplete multimodal HGR. The proposed AiFusion provides a promising solution to realize effective and robust multimodal HGR-based interfaces.

show abstract

FEEL: FEderated LEarning Framework for ELderly Healthcare Using Edge-IoMT

Ghosh

2023

IEEE Trans. Comput. Soc. Syst.

View full text Add to dashboard Cite

Hierarchical Progressive Network for Multimodal Medical Image Fusion in Healthcare Systems

Cited by 5 publications

References 62 publications

FedHealthFog: A federated learning-enabled approach towards healthcare analytics over fog computing platform

FedHealthFog: A federated learning-enabled approach towards healthcare analytics over fog computing platform

Alignment-Enhanced Interactive Fusion Model for Complete and Incomplete Multimodal Hand Gesture Recognition

FEEL: FEderated LEarning Framework for ELderly Healthcare Using Edge-IoMT

Contact Info

Product

Resources

About