CMC2R: Cross‐modal collaborative contextual representation for RGBT tracking

Abstract

The key challenge in RBGT tracking is how to fuse dual-modality information to build a robust RGB-T tracker. Motivated by CNN structure for local features, and visual transformer structure for global representations, the authors propose a two-stream hybrid structure, termed CMC 2 R, to take advantage of convolutional operations and self-attention mechanisms to lean the enhanced representation. CMC 2 R fuses local features and global representations under different resolutions through the transformer layer of the encoder block, and the two modalities are collaborated to get contextual information by the spatial and channel self-attention. The temporal association is performed with the track query, each track query models the entire track of an object, and updated frame-by-frame to build the long-range temporal relation. Experimental results show the effectiveness of the proposed method, and achieve the SOTAs performance.

Visible and infrared images fusion target tracking based on deep learning, progress, and prospects

“…According to the published literature, the first to introduce Transformer into the field of visible and infrared images fusion target tracking is a two-stream hybrid structure called CMC2R(Cross-modal collaborative contextual representation) [25]. The local and global representations at different resolutions were fused through the encoder block transformation layer, and the two modalities cooperated to acquire contextual information through the spatial and channel self-attention.…”