Deep Mutual Information Maximin for Cross-Modal Clustering

Abstract

Cross-modal clustering (CMC) aims to enhance the clustering performance by exploring complementary information from multiple modalities. However, the performances of existing CMC algorithms are still unsatisfactory due to the conflict of heterogeneous modalities and the high-dimensional non-linear property of individual modality. In this paper, a novel deep mutual information maximin (DMIM) method for cross-modal clustering is proposed to maximally preserve the shared information of multiple modalities while eliminating the superfluous information of individual modalities in an end-to-end manner. Specifically, a multi-modal shared encoder is firstly built to align the latent feature distributions by sharing parameters across modalities. Then, DMIM formulates the complementarity of multi-modalities representations as an mutual information maximin objective function, in which the shared information of multiple modalities and the superfluous information of individual modalities are identified by mutual information maximization and minimization respectively. To solve the DMIM objective function, we propose a variational optimization method to ensure it converge to a local optimal solution. Moreover, an auxiliary overclustering mechanism is employed to optimize the clustering structure by introducing more detailed clustering classes. Extensive experimental results demonstrate the superiority of DMIM method over the state-of-the-art cross-modal clustering methods on IAPR-TC12, ESP-Game, MIRFlickr and NUS-Wide datasets.

Reconsidering Representation Alignment for Multi-view Clustering

“…In recent years, deep learning architectures have seen widespread adoption in MVC, resulting in the deep MVC subfield. Methods developed within this subfield have shown state-of-the-art clustering performance on several multi-view datasets [1][2][3][4][5][6], largely outperforming traditional, non-deeplearning-based methods [1]. Despite these promising developments, we identify significant drawbacks with the current state of the field.…”

“…Despite these promising developments, we identify significant drawbacks with the current state of the field. Selfsupervised learning (SSL) is a crucial component in many recent methods for deep MVC [1][2][3][4][5][6]. However, the large number of methods, all with unique components and arguments about how they work, makes it challenging to identify clear directions and trends in the development of new components and methods.…”

Zero-shot-Learning Cross-Modality Data Translation Through Mutual Information Guided Stochastic Diffusion

“…MI is useful in cross-modality data processing tasks as the statistical features are assumed to be identical. It has been applied to tackle many unsupervised learning problems such as cross-modality data retrieval [25], data representations [22,74,17], domain adaptation [42], and cross-modal clustering [40] etc.. A particular case of MI is using MI for measuring a random variable itself: M I(X, X), which is called Entropy.…”

Low-rank tensor approximation with local structure for multi-view intrinsic subspace clustering

“…Wang et al [30] utilized the subgraph-level summary to build an effective mutual information estimator, which was optimized to strengthen the robustness of graph representation. Mao et al [31] explored the shared information across modalities via maximizing the mutual information between them. Schnapp et al [32] selected important features with minimum mutual information with labels.…”