MHTN: Modal-Adversarial Hybrid Transfer Network for Cross-Modal Retrieval

Abstract

Cross-modal retrieval has drawn wide interest for retrieval across different modalities of data (such as text, image, video, audio and 3D model). However, existing methods based on deep neural network (DNN) often face the challenge of insufficient cross-modal training data, which limits the training effectiveness and easily leads to overfitting. Transfer learning is usually adopted for relieving the problem of insufficient training data, but it mainly focuses on knowledge transfer only from large-scale datasets as single-modal source domain (such as ImageNet) to single-modal target domain. In fact, such large-scale single-modal datasets also contain rich modal-independent semantic knowledge that can be shared across different modalities. Besides, large-scale cross-modal datasets are very labor-consuming to collect and label, so it is significant to fully exploit the knowledge in singlemodal datasets for boosting cross-modal retrieval. To achieve this goal, this paper proposes modal-adversarial hybrid transfer network (MHTN), which to the best of our knowledge is the first work to realize knowledge transfer from single-modal source domain to cross-modal target domain, and learn cross-modal common representation. It is an end-to-end architecture with two subnetworks: (1) Modal-sharing knowledge transfer subnetwork is proposed to jointly transfer knowledge from a large-scale singlemodal dataset in source domain to all modalities in target domain with a star network structure, which distills modal-independent supplementary knowledge for promoting cross-modal common representation learning. (2) Modal-adversarial semantic learning subnetwork is proposed to construct an adversarial training mechanism between common representation generator and modality discriminator, making the common representation discriminative for semantics but indiscriminative for modalities to enhance crossmodal semantic consistency during transfer process. Comprehensive experiments on 4 widely-used datasets show its effectiveness and generality.

A New Benchmark and Approach for Fine-grained Cross-media Retrieval

“…I→T I→A I→V T→I T→A T→V A→I A→T A→V V→I V→T V→A Average Our FGCrossNet 0.210 0.526 0.606 0.255 0.181 0.208 0.553 0.159 0.443 0.629 0.195 0.437 0.366 MHTN [20] 0.116 0.195 0.281 0.124 0.138 0.185 0.196 0.127 0.290 0.306 0.186 0.306 0.204 ACMR [21] 0.162 0.119 0.477 0.075 0.015 0.081 0.128 0.028 0.068 0.536 0.138 0.111 0.162 JRL [22] 0.160 0.085 0.435 0.190 0.028 0.095 0.115 0.035 0.065 0.517 0.126 0.068 0.160 GSPH [23] 0.140 0.098 0.413 0.179 0.024 0.109 0.129 0.024 0.073 0.512 0.126 0.086 0.159 CMDN [24] 0.099 0.009 0.377 0.123 0.007 0.078 0.017 0.008 0.010 0.446 0.081 0.009 0.105 SCAN [25] 0.050…”

Adversarial Cross-Modal Retrieval via Learning and Transferring Single-Modal Similarities

“…On the other hand, the cross-modal hashing methods mainly focus on the retrieval efficiency by mapping the items of different modalities into a common binary Hamming space. Benefited from the strong ability of distribution modeling and discriminative representation learning, some recent crossmodal retrieval methods have collaborated with GAN models [9,10,2]. In this work, our method also follows the similar adversarial learning framework that uses the single-modal similarities to guide the cross-modal representation learning.…”

“…The ACMR [2] method proposes the triplet loss and the modality classifier for preserving the modality level semantic struc-tures. The MHTN [10] is proposed to minimize the maximum mean discrepancy between modalities, which preserves more flexibility for the generator to project vectors into a new space. The difference between CMST and the previous work is that CMST can learn the item-level semantic relationships between unpaired items in an unsupervised way.…”

Diachronic Cross-modal Embeddings

“…Until now, these embeddings have been learned in a static manner, i.e. without preserving the time dimension, and thus ignoring the evolution of modality interactions [5,7,12,20,21,23,27,30,31,35,36]. Approaches have ranged from solutions that organize the space according to linear correlations [23,33,36] (image and texts cooccurrence), semantic [20,30,34,35] (category information) and/or temporal correlations [27].…”