Multimodal Deep Generative Models for Remote Medical Applications

Abstract

Visible-to-Thermal (VT) face translation is an under-studied problem of image-to-image translation that offers an AI-enabled alternative to traditional thermal sensors. Over three phases, my Doctoral Proposal explores developing multimodal deep generative solutions that can be applied towards telemedicine applications. These include the contribution of a novel Thermal Face Contrastive GAN (TFC-GAN), exploration of hybridized diffusion-GAN models, application on real clinical thermal data at the National Institutes of Health, and exploration of strategies for Federated Learning (FL) in heterogenous data settings.

Prompting Multi-Modal Image Segmentation with Semantic Grouping

“…The effective information that different modalities can provide differs due to various features across domains, e.g., acquisition frequency, visual disparity, etc. However, aggregation-based fusion tends to ignore intra-modal propagation, resulting in insufficient balance between inter-modal knowledge sharing and intra-modal information processing (Xu et al 2017;Gao et al 2021;Ordun 2023).…”