Error-Corrected Margin-Based Deep Cross-Modal Hashing for Facial Image Retrieval

Abstract

Cross-modal hashing facilitates mapping of heterogeneous multimedia data into a common Hamming space, which can be utilized for fast and flexible retrieval across different modalities. In this paper, we propose a novel cross-modal hashing architecture-deep neural decoder cross-modal hashing (DNDCMH), which uses a binary vector specifying the presence of certain facial attributes as an input query to retrieve relevant face images from a database. The DNDCMH network consists of two separate components: an attribute-based deep cross-modal hashing (ADCMH) module, which uses a margin (m)-based loss function to efficiently learn compact binary codes to preserve similarity between modalities in the Hamming space, and a neural error correcting decoder (NECD), which is an error correcting decoder implemented with a neural network. The goal of NECD network in DNDCMH is to error correct the hash codes generated by ADCMH to improve the retrieval efficiency. The NECD network is trained such that it has an error correcting capability greater than or equal to the margin (m) of the margin-based loss function. This results in NECD can correct the corrupted hash codes generated by ADCMH up to the Hamming distance of m. We have evaluated and compared DNDCMH with state-of-the-art cross-modal hashing methods on standard datasets to demonstrate the superiority of our method.! DCMH [22], the inter-modal triplet embedding loss encourages the heterogeneous correlation across different modalities, and the intra-modal triplet loss encodes the discriminative power of the hash codes. Moreover, a regularization loss is used to apply adjacency consistency to ensure that the hash codes can keep the original similarities in Hamming space. However, in margin-based loss functions, some of the instances of different modalities of the same subject may not be close enough in Hamming space to guarantee all the correct retrievals. Therefore, it is important to bring the different modalities of the same subject closer to each other in Hamming space to improve the retrieval efficiency.In this work, we observe that in addition to the regular DCMH techniques [13], [24], [25], which exploit entropy maximization and quantization losses in the objective function of the DCMH, an error-correcting code (ECC) decoder can be used as an additional component to compensate for the heterogeneity gap and reduce the Hamming distance between the different modalities of the same subject in order to improve the cross-modal retrieval efficiency. We presume that the hash code generated by DCMH is a binary vector that is within a certain distance from a codeword of an ECC. When the hash code generated by DCMH is passed through an ECC decoder, the closest codeword to this hash code is found, which can be used as a final hash code for the retrieval process. In this process, the attribute hash code and image hash code of the same subject are forced to map to the same codeword, thereby reducing the distance of the corresponding hash codes. This brings more relevant facial images ...

A Cross-Modal Image and Text Retrieval Method Based on Efficient Feature Extraction and Interactive Learning CAE

“…e premise of hash transformation is that the hash codes of similar samples are also similar. Reference [22] proposed a method called DNDCMH. is algorithm uses binary vectors specifying the existence of specific facial attributes as input queries to retrieve relevant facial images from the database.…”

“…e corresponding research is a multilabel kernel canonical correlation analysis (ml-KCCA) method proposed in [15] and a cross-modal hashing retrieval method (DNDCMH) proposed in [22]. In addition, in order to highlight the effectiveness of the interactive learning CAE model proposed in this paper, it is compared with other methods based on the CAE model, such as the text retrieval method based on multimodal semantic automatic encoder (SCAE) proposed in [28].…”

“…e image-text retrieval results obtained by the proposed method and [22] retrieval method are shown in Table 1. It is the text retrieval result of the image on the Flickr30K test set.…”

“…ese visually presented phenomena more intuitively illustrate the effectiveness of the proposed method. In [22], DNDCMH is used to achieve text retrieval. Due to the lack of image feature extraction effect, the correct text is less.…”

Transporting Labels via Hierarchical Optimal Transport for Semi-Supervised Learning

“…Recent developments in CNNs have provided promising results for many applications in machine learning and computer vision such as facial recognition [1,2,3,4,5,6], image retrieval [4,7,8,9], image generation [10,11,10], and adversarial attack [12,13]. However, the success of CNN models requires a vast amount of well-annotated training data, which is not always feasible to perform manually [14,15].…”