Graph Convolutional Multi-modal Hashing for Flexible Multimedia Retrieval

Abstract

Multi-modal hashing makes an important contribution to multimedia retrieval, where a key challenge is to encode heterogeneous modalities into compact hash codes. To solve this dilemma, graphbased multi-modal hashing methods generally define individual affinity matrix of each independent modality and apply linear algorithm for heterogeneous modalities fusion and compact hash learning. Several other methods construct graph Laplacian matrix based on semantic information to help learn discriminative hash code. However, these conventional methods roughly ignore the structural similarity of training set and the complex relations among multi-modal samples, which leads to unsatisfactory complementarity of fused hash codes. More notably, they are faced with two other important problems: huge computing and storage costs caused by graph construction and partial modality feature lost problem when incomplete query sample comes. In this paper, we propose a Flexible Graph Convolutional Multi-modal Hashing (FGCMH) method that adopts GCNs with linear complexity to preserve both the modality-individual and modality-fused structural similarity for discriminative hash learning. Necessarily, accurate multimedia retrieval can be performed on complete and incomplete datasets with our method. Specifically, multiple modality-individual GCNs under semantic guidance are proposed to act on each individual modality independently for intra-modality similarity preserving, then the output representations are fused into a fusion graph with adaptive weighting scheme. Hash GCN and semantic GCN, which share parameters in the first two layers, propagate fusion information and generate hash codes under high-level label space supervision. In the query stage, our method adaptively captures various multi-modal contents in a flexible and robust way, even if partial modality features are lost. Experimental results on three publicly datasets show the flexibility and effectiveness of our proposed method.

CLIP-Based Adaptive Graph Attention Network for Large-Scale Unsupervised Multi-Modal Hashing Retrieval

“…In particular, graph convolutional network hashing (GCNH) [ 38 ] introduces an asymmetric graph convolution layer that addresses the problems of scalability and out-of-sample extension when exploiting affinity graphs for hashing. Graph convolutional multi-modal hashing (GCMH) [ 39 ] proposes multiple modality-individual GCNs under semantic guidance to act on each modality independently to preserve intra-modality similarity, and then fuse the output representations into a fusion graph with an adaptive weighting scheme. Aggregation-based graph convolutional hashing (AGCH) [ 36 ] designed an elegant aggregation strategy that leverages multiple similarity measures to build an accurate semantic similarity matrix and employs graph convolutional neural networks to aggregate similarity information across modal data, which further mines the semantic relevance of different modal data.…”

Gaussian similarity preserving for cross-modal hashing

“…However, those pre-defined graphs cannot be adaptively learned during the training to better express the semantic structures. Different from the label-based reliable static association graph in the supervised methods [6,29], the "static graph" constructed based on the original data measurement in the unsupervised methods [25,46] actually introduces the bias in the original feature measurement.…”