Towards a Unified Middle Modality Learning for Visible-Infrared Person Re-Identification

Abstract

Visible-infrared person re-identification (VI-ReID) aims to search identities of pedestrians across different spectra. In this task, one of the major challenges is the modality discrepancy between the visible (VIS) and infrared (IR) images. Some state-of-the-art methods try to design complex networks or generative methods to mitigate the modality discrepancy while ignoring the highly non-linear relationship between the two modalities of VIS and IR. In this paper, we propose a non-linear middle modality generator (MMG), which helps to reduce the modality discrepancy. Our MMG can effectively project VIS and IR images into a unified middle modality image (UMMI) space to generate middle-modality (M-modality) images. The generated M-modality images and the original images are fed into the backbone network to reduce the modality discrepancy. Furthermore, in order to pull together the two types of M-modality images generated from the VIS and IR images in the UMMI space, we propose a distribution consistency loss (DCL) to make the modality distribution of the generated M-modalities images as consistent as possible. Finally, we propose a middle modality network (MMN) to further enhance the discrimination and richness of features in an explicit manner. Extensive experiments have been conducted to validate the superiority of MMN for VI-ReID over some state-of-theart methods on two challenging datasets. The gain of MMN is more than 11.1% and 8.4% in terms of Rank-1 and mAP, respectively, even compared with the latest state-of-the-art methods on the SYSU-MM01 dataset.

Towards Homogeneous Modality Learning and Multi-Granularity Information Exploration for Visible-Infrared Person Re-Identification

“…[44] propose a bi-directional top-ranking loss, which samples positive and negative pairs from different modalities and optimizes such cross-modality triplets with a bi-directional interactive iteration manner. More recently, some other works adopt adversarial training strategies to reduce the cross-modality distribution divergence in image-level [29], [30], [32], [36], [46], [49]. For a instance, they transfer stylistic properties of visible images to their infrared counterpart, with an identitypreserving constraint [30], [32] or cycle consistency [29], [36].…”

Person Re-Identification with RGB–D and RGB–IR Sensors: A Comprehensive Survey

“…Apart from modality-translation-based Re-ID approaches, there are a few attempts [ 95 , 96 , 97 , 98 ] that introduce a third modality to reduce the modality discrepancy. The idea of using a third modality was proposed by Li et al [ 95 ], who introduced an “X” modality as a middle modality to eliminate cross-modal discrepancies.…”

“…Following the same pipeline, in [ 96 , 97 ], real images from both modalities were combined with ground-truth labels to generate third-modal images, which help to reduce modality-related biases. In [ 98 ], a non-linear middle modality generator was proposed that effectively projects images from both modalities onto a unified space to generate an additional modality to reduce the modality discrepancies.…”

Margin-Based Modal Adaptive Learning for Visible-Infrared Person Re-Identification

“…The most popular architecture [ 5 , 35 , 48 ] is a double-stream deep network, where shallow layers are independent for learning modal-specific features and deep layers are shared for learning modal-common features. Some researchers improved the double-stream architecture via fine part alignment designs [ 40 , 49 ], attention mechanisms [ 35 , 36 ], or new neural structures, such as graph [ 27 ] and transformer [ 32 , 50 ].…”