Self-supervised Disentanglement of Modality-Specific and Shared Factors Improves Multimodal Generative Models

Abstract

Multimodal generative models learn a joint distribution over multiple modalities and thus have the potential to learn richer representations than unimodal models. However, current approaches are either inefficient in dealing with more than two modalities or fail to capture both modality-specific and shared variations. We introduce a new multimodal generative model that integrates both modality-specific and shared factors and aggregates shared information across any subset of modalities efficiently. Our method partitions the latent space into disjoint subspaces for modality-specific and shared factors and learns to disentangle these in a purely self-supervised manner. Empirically, we show improvements in representation learning and generative performance compared to previous methods and showcase the disentanglement capabilities. Electronic supplementary material The online version of this chapter (10.1007/978-3-030-71278-5_33) contains supplementary material, which is available to authorized users.

A survey of multimodal deep generative models

“…Furthermore, the stability of their learning remains a challenge, such as mode collapse, which is confirmed to be more serious in a multimodal setting [113]. Therefore, VAEs have become the mainstream in multimodal deep generative models, where GANs are sometimes used to improve the quality of the generation of multimodal VAEs [92,113] or to implement divergence between distributions in VAEs [15]. Table 1.…”

“…× × (memory cost) MVAE [112] ( ) (require sub-sampling) × MMVAE [79] × (MoE) × mmJSD [88] × (MoE) × mmJSD (MS) [88] × (MoE) AVAE [118] × × MoPoE-VAE [89] × × (computational cost) PVAE [33] × (memory cost) DMVAE [48] DMVAE [15] MFM [99] × (memory cost) [78] × (MoE) ×…”

“…Wu et al [112] also proposed a method for a weakly-supervised learning setting where all modalities are not present at training, followed by Shi et al [78]. Modality-specific latent variables were first introduced into JVAE by Hsu et al [33] and Tsai et al [99], and then models with various inference methods and objectives were proposed [15,48,88].…”

“…Daunhawer et al [15] introduce the following two objective in addition to the objective of JVAE. The first objective is the mutual information between the multimodal input X and the modality-invariant representation c, which imposes the inclusion of multimodal information in c. However, since it is difficult to compute this mutual information directly, they estimate its lower bound using the sample-based InfoNCE estimator [65].…”

On the Limitations of Multimodal VAEs

“…Similar to previous work, we have only considered models with simple priors, such as Gauss and Laplace distributions with independent dimensions. Further, we have not considered models with modality-specific latent spaces, which seem to yield better empirical results (Hsu and Glass, 2018;Sutter et al, 2020;Daunhawer et al, 2020), but currently lack theoretical grounding. Modality-specific latent spaces offer a potential solution to the problem of cross-modal prediction by providing modality-specific context from the target modalities to each decoder.…”