Harmonized Multimodal Learning with Gaussian Process Latent Variable Models

Abstract

Multimodal learning aims to discover the relationship between multiple modalities. It has become an important research topic due to extensive multimodal applications such as cross-modal retrieval. This paper attempts to address the modality heterogeneity problem based on Gaussian process latent variable models (GPLVMs) to represent multimodal data in a common space. Previous multimodal GPLVM extensions generally adopt individual learning schemes on latent representations and kernel hyperparameters, which ignore their intrinsic relationship. To exploit strong complementarity among different modalities and GPLVM components, we develop a novel learning scheme called Harmonization, where latent model parameters are jointly learned from each other. Beyond the correlation fitting or intra-modal structure preservation paradigms widely used in existing studies, the harmonization is derived in a model-driven manner to encourage the agreement between modality-specific GP kernels and the similarity of latent representations. We present a range of multimodal learning models by incorporating the harmonization mechanism into several representative GPLVM-based approaches. Experimental results on four benchmark datasets show that the proposed models outperform the strong baselines for cross-modal retrieval tasks, and that the harmonized multimodal learning method is superior in discovering semantically consistent latent representation.

Feature Integration via Semi-Supervised Ordinally Multi-Modal Gaussian Process Latent Variable Model

“…In this paper, Eq. ( 18) can be solved by the scaled conjugate gradient (SCG) technique [24] in the same manner as previous GPLVM-based approaches [22,25]. Since the proposed method assumes that the similarity matrices of the observations are generated from the latent variables in Eq.…”

Cross-Modal Hashing by lp -Norm Multiple Subgraph Combination

“…Each image is also annotated by 5 independent sentences via Amazon Mechanical Turk. Following [54], after removing images without labels, we randomly select 10,000 image-text pairs for testing, and the remaining 72,081 pairs are used for training. For image representation, we use the multi-layer CNN features extracted from conv 4 , conv 5 , fc 6 and fc 7 of a pre-trained VGG-19 model.…”

“…(2) Harmonized GPLVM [54]: a harmonized multi-modal GPLVM which performs topological alignment between the hyperparameter space of multi-modal GPLVM and the kernel matrix of the joint latent space. We report the results using the best variants hm-SimGP (tr) and hm-RSimGP (tr) with tracenorm kernel alignment.…”

Gaussian process linking functions for mind, brain, and behavior

“…The GPJM framework is based on Gaussian process latent variable models (GPLVMs) and their hierarchical extensions (22)(23)(24)(25), which pursue commonly shared latent representations across simultaneous observations from multiple measurement sources. GPLVMs have been successfully used in many machine-learning applications (26)(27)(28)(29)(30)(31).…”