Learning Shared Semantic Space with Correlation Alignment for Cross-Modal Event Retrieval

Abstract

In this paper, we propose to learn shared semantic space with correlation alignment (S 3 CA) for multimodal data representations, which aligns nonlinear correlations of multimodal data distributions in deep neural networks designed for heterogeneous data. In the context of cross-modal (event) retrieval, we design a neural network with convolutional layers and fully-connected layers to extract features for images, including images on Flickr-like social media. Simultaneously, we exploit a fully-connected neural network to extract semantic features for texts, including news articles from news media. In particular, nonlinear correlations of layer activations in the two neural networks are aligned with correlation alignment during the joint training of the networks. Furthermore, we project the multimodal data into a shared semantic space for cross-modal (event) retrieval, where the distances between heterogeneous data samples can be measured directly. In addition, we contribute a Wiki-Flickr Event dataset, where the multimodal data samples are not describing each other in pairs like the existing paired datasets, but all of them are describing semantic events. Extensive experiments conducted on both paired and unpaired datasets manifest the effectiveness of S 3 CA, outperforming the state-of-the-art methods.

Hybrid SOM based cross-modal retrieval exploiting Hebbian learning

“…The technique is known as hybrid representation learning (HRL) in which stacked restricted Boltzmann machines are used for extracting modality-friendly representation and a multi-modal deep belief network is exploited for extracting modality-mutual representation. Shared semantic space with correlation alignment (S 3 CA) is introduced in [26] for multi-modal data representation. Non-linear correlations of multi-modal data distributions are aligned in deep neural networks constructed for dissimilar data.…”

Comparative analysis on cross-modal information retrieval: A review

“…This whole network facilitates effective iterative parameter optimization. In [99], a shared se-mantic space with correlation alignment (S3CA) is proposed for cross-modal data representation. It aligns the non-linear correlations of cross-modal data distribution in deep neural networks made for diversified data.…”

Boundary Preserving Twin Energy-Based-Models for Image to Image Translation

“…CFDNet [6] proposed to minimize the distance between the characteristic functions of distribution for feature alignment. On similar lines, [40] proposed to learn a latent space by matching the correlation matrix of different domains. All these works, however, require prior knowledge of ground truth labels which may not be available in most scenarios.…”