MSO: Multi-Feature Space Joint Optimization Network for RGB-Infrared Person Re-Identification

Abstract

The RGB-infrared cross-modality person re-identification (ReID) task aims to recognize the images of the same identity between the visible modality and the infrared modality. Existing methods mainly use a two-stream architecture to eliminate the discrepancy between the two modalities in the final common feature space, which ignore the single space of each modality in the shallow layers. To solve it, in this paper, we present a novel multi-feature space joint optimization (MSO) network, which can learn modality-sharable features in both the single-modality space and the common space. Firstly, based on the observation that edge information is modality-invariant, we propose an edge features enhancement module to enhance the modality-sharable features in each single-modality space. Specifically, we design a perceptual edge features (PEF) loss after the edge fusion strategy analysis. According to our knowledge, this is the first work that proposes explicit optimization in the singlemodality feature space on cross-modality ReID task. Moreover, to increase the difference between cross-modality distance and class distance, we introduce a novel cross-modality contrastive-center (CMCC) loss into the modality-joint constraints in the common feature space. The PEF loss and CMCC loss jointly optimize the model in an end-to-end manner, which markedly improves the network's performance. Extensive experiments demonstrate that the proposed model significantly outperforms state-of-the-art methods on both the SYSU-MM01 and RegDB datasets. CCS CONCEPTS• Computing methodologies → Visual content-based indexing and retrieval.

Feature Fusion and Center Aggregation for Visible-Infrared Person Re-Identification

“…These SOTA approaches contain Hi-CMD [48] adopting ID-discriminative factors to robust cross-modality match,JSIA [49], CoAL [54], and DF 2 AM [56]using two-level alignment approaches for cross-modality person matching task, HC [47] using multiple loss functions (enumerate angular triplet (EAT) loss, hetero-center loss, and cross-modality knowledge distillation (CMKD) loss) to enhance the feature distinctiveness, CMSP [51] and ATTRI [52] adopting extra constraints to increase intra-class cross-modality similarity while mitigating modality-specific information, HAT [53] generating additional modality between both visible and infrared modalities for alleviating the modality differences, NFS [55] utilizing a BN-oriented/feature search space to achieve standard optimization/automatic feature selection for the cross-modality work. MSO [67] proposed a perceptual edge features (PEF) loss to optimize their network. G2DA [68] proposed a Geometry-Guided Dual-Alignment method to reduce crossmodality differences between part semantics and structural relations.…”

Modality Discrepancy Reduction for Visible-Infrared Person Re-Identification under Complete Modality Missing

“…When color information is not reliable for pedestrian matching, shape information serves as a complement option [47][48][49]. Moreover, with the shape representation as input of the network, the modality discrepancy is also reduced [47]. However, compared with RGB or IR images as input of the network, shape representation greatly loses identity-related information which is also inferior for person re-identification task.…”

“…For visible-infrared person re-identification problem, due to the difference of infrared images and visible images, color, as a common cue for person re-identification task, is not reliable. When color information is not reliable for pedestrian matching, shape information serves as a complement option [47][48][49]. Moreover, with the shape representation as input of the network, the modality discrepancy is also reduced [47].…”

Joint Modal Alignment and Feature Enhancement for Visible-Infrared Person Re-Identification

“…The representation learning methods [8,11] focus on how to improve the feature extraction process in both modals. Metric learning methods [16,18,19] focus on modifying the loss function during model training to optimize the feature metric space. Both representation learning methods and metric learning methods attempt to align modals during the feature-learning phase, but often have serious artificial design traces and bring complex network structures and calculations, which reduce the transferability of the model.…”