Deep Adaptive Fusion Network for High Performance RGBT Tracking

Gao, Yimin; Li, Chenglong; Zhu, Yabin; Tang, Jin; He, Tao; Wang, Feng

doi:10.1109/iccvw.2019.00017

Cited by 111 publications

(52 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The backbone network of the CEDiMP tracking framework is ResNet50 [ 58 ], but only the first 4 blocks are used. In order to make the feature representation model of the tracker obtain the powerful representation capabilities of multi-modal common features and single-modal unique features, we perform channel exchanging operations in the backbone network of the RGB and TIR modalities when completing feature extraction tasks.…”

Section: Methodsmentioning

confidence: 99%

Channel Exchanging for RGB-T Tracking

Zhao

Zhu

Ren

et al. 2021

Sensors

View full text Add to dashboard Cite

It is difficult to achieve all-weather visual object tracking in an open environment only utilizing single modality data input. Due to the complementarity of RGB and thermal infrared (TIR) data in various complex environments, a more robust object tracking framework can be obtained using video data of these two modalities. The fusion methods of RGB and TIR data are the core elements to determine the performance of the RGB-T object tracking method, and the existing RGB-T trackers have not solved this problem well. In order to solve the current low utilization of information intra single modality in aggregation-based methods and between two modalities in alignment-based methods, we used DiMP as the baseline tracker to design an RGB-T object tracking framework channel exchanging DiMP (CEDiMP) based on channel exchanging. CEDiMP achieves dynamic channel exchanging between sub-networks of different modes hardly adding any parameters during the feature fusion process. The expression ability of the deep features generated by our data fusion method based on channel exchanging is stronger. At the same time, in order to solve the poor generalization ability of the existing RGB-T object tracking methods and the poor ability in the long-term object tracking, more training of CEDiMP on the synthetic dataset LaSOT-RGBT is added. A large number of experiments demonstrate the effectiveness of the proposed model. CEDiMP achieves the best performance on two RGB-T object tracking benchmark datasets, GTOT and RGBT234, and performs outstandingly in the generalization testing.

show abstract

Section: Methodsmentioning

confidence: 99%

Channel Exchanging for RGB-T Tracking

Zhao

Zhu

Ren

et al. 2021

Sensors

View full text Add to dashboard Cite

show abstract

“…DAPNet [8] simultaneously integrates all the chosen layers and modalities by using an adaptive fusion module recursively, achieving feature pruning by global average pooling to reduce redundancy and noise. Further, DAFNet [49] integrates the adaptive fusion sub-network from [48] and realizes quality-aware fusion in each layer between the two modalities. To achieve fine-grained fusion, MANet [7] divides features into three categories (modality-specific, modality-shared, and instance-specific), exploiting a more exhaustive featurelevel fusion approach to get robust feature representation by multiple adaptors.…”

Section: Tracking With Multiple Modalitiesmentioning

confidence: 99%

“…A multi-branch architecture is employed in CAT [54] for target appearance modelling in allusion to the modalityspecific and modality-shared challenges, which totally count to 5. Unlike DAPNet [8] and DAFNet [49] keep only the fused features retained, TFNet [55] designs a trident architecture for better excavating the modality-specific information. Constraint to the limited RGBT data, the trackers aforementioned are all equipped with lightweight feature extractors.…”

Section: Tracking With Multiple Modalitiesmentioning

confidence: 99%

Exploring Fusion Strategies for Accurate RGBT Visual Object Tracking

Tang¹,

Xu²,

Li³

et al. 2022

Preprint

View full text Add to dashboard Cite

We address the problem of multi-modal object tracking in video and explore various options of fusing the complementary information conveyed by the visible (RGB) and thermal infrared (TIR) modalities including pixel-level, feature-level and decision-level fusion. Specifically, different from the existing methods, paradigm of image fusion task is heeded for fusion at pixel level. Featurelevel fusion is fulfilled by attention mechanism with channels excited optionally. Besides, at decision level, a novel fusion strategy is put forward since an effortless averaging configuration has shown the superiority. The effectiveness of the proposed decision-level fusion strategy owes to a number of innovative contributions, including a dynamic weighting of the RGB and TIR contributions and a linear template update operation. A variant of which produced the winning tracker at the Visual Object Tracking Challenge 2020 (VOT-RGBT2020). The concurrent exploration of innovative pixel-and feature-level fusion strategies highlights the advantages of the proposed decision-level fusion method. Extensive experimental results on three challenging datasets, i.e., GTOT, VOT-RGBT2019, and VOT-RGBT2020, demonstrate the effectiveness and robustness of the proposed method, compared to the state-of-the-art approaches. Code will be shared at https://github.com/Zhangyong-Tang/DFAT.

show abstract

“…DAPNet [15] achieves the fusion task by recurrently using a sub-network, which consists of a convolutional layer, a Rectified Linear Unit (ReLU) activation [16] and a normalization layer, at different feature levels. Compared with the coarse fusion sub-network used in DAPNet, DAFNet [17] further designs an adaptive fusion module which is similar to that in FANet. Carefully programmed, MANet [18] expects to extract the modality-specific, modality-shared and object-specific clues through modality, generator and instance adapters respectively.…”

Section: A Mdnet-based Rgbt Trackersmentioning

confidence: 99%

A Survey for Deep RGBT Tracking

Tang¹,

Xu²,

Wu³

2022

Preprint

View full text Add to dashboard Cite

Visual object tracking with the visible (RGB) and thermal infrared (TIR) electromagnetic waves, shorted in RGBT tracking, recently draws increasing attention in the tracking community. Considering the rapid development of deep learning, a survey for the recent deep neural network based RGBT trackers is presented in this paper. Firstly, we give brief introduction for the RGBT trackers concluded into this category. Then, a comparison among the existing RGBT trackers on several challenging benchmarks is given statistically. Specifically, MDNet and Siamese architectures are the two mainstream frameworks in the RGBT community, especially the former. Trackers based on MDNet achieve higher performance while Siamese-based trackers satisfy the real-time requirement. In summary, since the large-scale dataset LasHeR is published, the integration of end-to-end framework, e.g., Siamese and Transformer, should be further considered to fulfil the real-time as well as more robust performance. Furthermore, the mathematical meaning should be more considered during designing the network. This survey can be treated as a look-up-table for researchers who are concerned about RGBT tracking.

show abstract

Deep Adaptive Fusion Network for High Performance RGBT Tracking

Cited by 111 publications

References 16 publications

Channel Exchanging for RGB-T Tracking

Channel Exchanging for RGB-T Tracking

Exploring Fusion Strategies for Accurate RGBT Visual Object Tracking

A Survey for Deep RGBT Tracking

Contact Info

Product

Resources

About