Multi-Modal Fusion for End-to-End RGB-T Tracking

Abstract

We propose an end-to-end tracking framework for fusing the RGB and TIR modalities in RGB-T tracking. Our baseline tracker is DiMP (Discriminative Model Prediction), which employs a carefully designed target prediction network trained end-to-end using a discriminative loss. We analyze the effectiveness of modality fusion in each of the main components in DiMP, i.e. feature extractor, target estimation network, and classifier. We consider several fusion mechanisms acting at different levels of the framework, including pixel-level, feature-level and response-level. Our tracker is trained in an end-to-end manner, enabling the components to learn how to fuse the information from both modalities. As data to train our model, we generate a largescale RGB-T dataset by considering an annotated RGB tracking dataset (GOT-10k) and synthesizing paired TIR images using an image-to-image translation approach. We perform extensive experiments on VOT-RGBT2019 dataset and RGBT210 dataset, evaluating each type of modality fusing on each model component. The results show that the proposed fusion mechanisms improve the performance of the single modality counterparts. We obtain our best results when fusing at the feature-level on both the IoU-Net and the model predictor, obtaining an EAO score of 0.391 on VOT-RGBT2019 dataset. With this fusion mechanism we achieve the state-of-the-art performance on RGBT210 dataset.

The Seventh Visual Object Tracking VOT2019 Challenge Results

“…The mfDiMP tracker contains an end-to-end tracking framework for fusing the RGB and TIR modalities in RGB-T tracking [107]. The baseline tracker is DiMP (Discriminative Model Prediction) [8], which employs a carefully designed target prediction network trained end-to-end using a discriminative loss.…”

“…The baseline tracker is DiMP (Discriminative Model Prediction) [8], which employs a carefully designed target prediction network trained end-to-end using a discriminative loss. The mfDiMP tracker fuses modalities at the feature level on both the IoU predictor and the model predictor of DiMP [107]. SiamDW-T is based on our previous work [109], and extends it with two fusion strategies for RGBT tracking.…”

SiamCDA: Complementarity- and Distractor-Aware RGB-T Tracking Based on Siamese Network

“…The early RGB-T tracking algorithms [8]- [10] are based on some handcrafted features. With the development of deep learning, more and more RGB-T trackers based on deep features [12], [13], [26] are presented. These RGB-T trackers are usually designed on the basis of RGB trackers.…”

“…In [13], Li et al proposed a multi-adapter architecture to learn modality-shared, modality-specific, and instance-aware target representations, respectively. In addition, Zhang et al [26] introduced DiMP [33] as their baseline tracker and investigated different levels of fusion mechanisms to find the optimal fusion architecture. results demonstrated that their proposed fusion tracker significantly improved the performance of the baseline tracker with respect to unimodal tracking and achieved new state-of-the-art results.…”

“…(2) Multi-modal feature fusion For the RGB-T tracking task, how to effectively fuse the RGB and thermal information is one of the most important issues. Several methods have been proposed, such as element-wise summation [25], concatenation [26] and contentdependency weighting based fusion strategies [12] [17]. However, most of these existing fusion strategies do not consider the feature differences between the input RGB and thermal images during fusion.…”