Exploring Fusion Strategies for Accurate RGBT Visual Object Tracking

Abstract

We address the problem of multi-modal object tracking in video and explore various options of fusing the complementary information conveyed by the visible (RGB) and thermal infrared (TIR) modalities including pixel-level, feature-level and decision-level fusion. Specifically, different from the existing methods, paradigm of image fusion task is heeded for fusion at pixel level. Featurelevel fusion is fulfilled by attention mechanism with channels excited optionally. Besides, at decision level, a novel fusion strategy is put forward since an effortless averaging configuration has shown the superiority. The effectiveness of the proposed decision-level fusion strategy owes to a number of innovative contributions, including a dynamic weighting of the RGB and TIR contributions and a linear template update operation. A variant of which produced the winning tracker at the Visual Object Tracking Challenge 2020 (VOT-RGBT2020). The concurrent exploration of innovative pixel-and feature-level fusion strategies highlights the advantages of the proposed decision-level fusion method. Extensive experimental results on three challenging datasets, i.e., GTOT, VOT-RGBT2019, and VOT-RGBT2020, demonstrate the effectiveness and robustness of the proposed method, compared to the state-of-the-art approaches. Code will be shared at https://github.com/Zhangyong-Tang/DFAT.

RFN-Nest: An end-to-end residual fusion network for infrared and visible images

“…In the VOT2020-RGBT challenge, the core metric to evaluate the tracking performance is EAO. The four trackers used in the comparative experiment are: AFAT [44] with single modality (RGB and infrared) as input, combined RFN and AFAT (RFNT) [15], M2C2Frgbt [40], and the decisionlevel fusion tracker (DFAT) [43].…”

“…In order to apply our fusion framework to the multimodal object tracking task, a state-of-the-art RGBT tracker (DFAT) [43] is adopted as the baseline tracker. The DFAT won the third place in the evaluation on the public dataset and was the winning tracker in the VOT2020-RGBT challenge.…”

SCA-MMA: Spatial and Channel-Aware Multi-Modal Adaptation for Robust RGB-T Object Tracking

“…To employ the temporal continuity in a video sequence, the history information was integrated to obtain fusion features by computing the adaptive weights of previous frames [23]. Tang et al [24] proposed multiple fusion strategies from different perspectives (including pixel-level, feature-level and decision-level) to boost the performance of multi-modal object tracking in video.…”