Robust Latent Representations Via Cross-Modal Translation and Alignment

Abstract

Multi-modal learning relates information across observation modalities of the same physical phenomenon to leverage complementary information. Most multi-modal machine learning methods require that all the modalities used for training are also available for testing. This is a limitation when signals from some modalities are unavailable or severely degraded. To address this limitation, we aim to improve the testing performance of uni-modal systems using multiple modalities during training only. The proposed multi-modal training framework uses cross-modal translation and correlation-based latent space alignment to improve the representations of a worse performing (or weaker) modality. The translation from the weaker to the better performing (or stronger) modality generates a multi-modal intermediate encoding that is representative of both modalities. This encoding is then correlated with the stronger modality representation in a shared latent space. We validate the proposed framework on the AVEC 2016 dataset (RECOLA) for continuous emotion recognition and show the effectiveness of the framework that achieves state-ofthe-art (uni-modal) performance for weaker modalities.

Learning across diverse biomedical data modalities and cohorts: Challenges and opportunities for innovation

“…In a C 4 setting, colearning methods might be preferable due to their ability to function relatively well in scenarios where modalities may be missing during training or inference. 85 , 86 , 87 Colearning uses knowledge transfer from one modality to learn about a less-informed modality. Colearning methods include the utilization of multimodal embeddings, transfer learning, multitask learning, and generative networks, with each method aiding in mitigating real-world issues with multimodal data, such as missing modalities, noisy labels, and domain adaptation.…”