Multi-modal Mutual Topic Reinforce Modeling for Cross-media Retrieval

Abstract

As an important and challenging problem in the multimedia area, multi-modal data understanding aims to explore the intrinsic semantic information across different modalities in a collaborative manner. To address this problem, a possible solution is to effectively and adaptively capture the common cross-modal semantic information by modeling the inherent correlations between the latent topics from different modalities. Motivated by this task, we propose a supervised multi-modal mutual topic reinforce modeling (M 3 R) approach, which seeks to build a joint cross-modal probabilistic graphical model for discovering the mutually consistent semantic topics via appropriate interactions between model factors (e.g., categories, latent topics and observed multi-modal data). In principle, M 3 R is capable of simultaneously accomplishing the following two learning tasks: 1) modality-specific (e.g., image-specific or text-specific ) latent topic learning; and 2) cross-modal mutual topic consistency learning. By investigating the cross-modal topic-related distribution information, M 3 R encourages to disentangle the semantically consistent cross-modal topics (containing some common semantic information across different modalities). In other words, the semantically co-occurring cross-modal topics are reinforced by M 3 R through adaptively passing the mutually reinforced messages to each other in the modellearning process. To further enhance the discriminative power of the learned latent topic representations, M 3 R incorporates the auxiliary information (i.e., categories or labels) into the process of Bayesian modeling, which boosts the modeling capability of capturing the inter-class discriminative information. Experimental results over two benchmark datasets demonstrate the effectiveness of the proposed M 3 R in crossmodal retrieval.

Cross-media analysis and reasoning: advances and directions

“…For example, Roller and Schulte im Walde (2013) integrated visual features into latent Dirichlet allocation (LDA) and proposed a multimodal LDA model to learn representations for textual and visual data. Wang Y et al (2014) proposed a scheme called the multimodal mutual topic reinforce model (M 3 R), which seeks to discover mutually consistent semantic topics via appropriate interactions between model factors. These schemes represent data as topic distributions, and similarities are measured by the likelihood of observed data in terms of latent topics.…”

Multimedia retrieval based on non-linear graph-based fusion and partial least squares regression

“…An example is the well-known Latent Dirichlet Allocation (LDA). In [32], a supervised multimodal mutual topic reinforce modeling approach for cross-media retrieval, called M3R, is proposed. Some other methodologies are Partial Least Squares (PLS) and correlation matching.…”

Similarity Gaussian Process Latent Variable Model for Multi-modal Data Analysis

“…However, CCA-based methods lack probabilistic interpretation on the intra-modal similarities. Topic models [3,12,25] learn latent topics to describe the intrinsic semantic correlations in multi-modal data. Based on Latent Dirichlet Allocation (LDA) [3], a variety of constraints are imposed.…”

“…For example, CCA-based models [1,11,19,20] assume that the inter-modal relation is expressed by co-occurrence of multi-modal data objects. The inter-modal relation is also encoded as binary observation matrix to be fit by the correlation models [3,12,25,27]. By contrast, we directly impose two kinds of inter-modal relations (i.e., semantic similarity and dissimilarity) as smooth priors on the output of multi-modal GPLVM.…”