Thermal infrared and visible sequences fusion tracking based on a hybrid tracking framework with adaptive weighting scheme

Luo, Chengwei; Sun, Bin; Yang, Ke; Lu, Taoran; Yeh, Wei-Chang

doi:10.1016/j.infrared.2019.04.017

Cited by 53 publications

(19 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Besides, there are some low-level topics which regard the RGBT tracking as a downstream task to verify their robustness and generalization like RFN-Nest [56] Lastly, seldom trackers pay attention to the fusion at the decision level. [57] and [14] use KL Divergence to estimate the reliability and achieve adaptive fusion after the response maps of both modalities are available. [6] sums up the scores from different modalities roughly and only obtains similar results compared with pixel-level fusion.…”

Section: Tracking With Multiple Modalitiesmentioning

confidence: 99%

Exploring Fusion Strategies for Accurate RGBT Visual Object Tracking

Tang¹,

Xu²,

Li³

et al. 2022

Preprint

View full text Add to dashboard Cite

We address the problem of multi-modal object tracking in video and explore various options of fusing the complementary information conveyed by the visible (RGB) and thermal infrared (TIR) modalities including pixel-level, feature-level and decision-level fusion. Specifically, different from the existing methods, paradigm of image fusion task is heeded for fusion at pixel level. Featurelevel fusion is fulfilled by attention mechanism with channels excited optionally. Besides, at decision level, a novel fusion strategy is put forward since an effortless averaging configuration has shown the superiority. The effectiveness of the proposed decision-level fusion strategy owes to a number of innovative contributions, including a dynamic weighting of the RGB and TIR contributions and a linear template update operation. A variant of which produced the winning tracker at the Visual Object Tracking Challenge 2020 (VOT-RGBT2020). The concurrent exploration of innovative pixel-and feature-level fusion strategies highlights the advantages of the proposed decision-level fusion method. Extensive experimental results on three challenging datasets, i.e., GTOT, VOT-RGBT2019, and VOT-RGBT2020, demonstrate the effectiveness and robustness of the proposed method, compared to the state-of-the-art approaches. Code will be shared at https://github.com/Zhangyong-Tang/DFAT.

show abstract

Section: Tracking With Multiple Modalitiesmentioning

confidence: 99%

Exploring Fusion Strategies for Accurate RGBT Visual Object Tracking

Tang¹,

Xu²,

Li³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…For example, Laguela et al [64] researched how to generate a high-quality thermographic image of a building envelope by fusing infrared data with RGB images. Similarly, Luo et al's [65] hybrid tracking framework is an early fusion method. Conversely, Li et al's [66] two-stream CNN of RGB-thermal object tracking is a typical late fusion method.…”

Section: Rgb-depth Fusionmentioning

confidence: 99%

An Approach to Semantically Segmenting Building Components and Outdoor Scenes Based on Multichannel Aerial Imagery Datasets

et al. 2021

View full text Add to dashboard Cite

As-is building modeling plays an important role in energy audits and retrofits. However, in order to understand the source(s) of energy loss, researchers must know the semantic information of the buildings and outdoor scenes. Thermal information can potentially be used to distinguish objects that have similar surface colors but are composed of different materials. To utilize both the red–green–blue (RGB) color model and thermal information for the semantic segmentation of buildings and outdoor scenes, we deployed and adapted various pioneering deep convolutional neural network (DCNN) tools that combine RGB information with thermal information to improve the semantic and instance segmentation processes. When both types of information are available, the resulting DCNN models allow us to achieve better segmentation performance. By deploying three case studies, we experimented with our proposed DCNN framework, deploying datasets of building components and outdoor scenes, and testing the models to determine whether the segmentation performance had improved or not. In our observation, the fusion of RGB and thermal information can help the segmentation task in specific cases, but it might also make the neural networks hard to train or deteriorate their prediction performance in some cases. Additionally, different algorithms perform differently in semantic and instance segmentation.

show abstract

“…Because the fusion of RGB and TIR data more easily achieves all-weather object tracking in the open environment, the researches on RGB-T object tracking methods become more and more popular. From the perspective of data fusion, the RGB-T object tracking framework can be roughly divided into traditional methods [ 38 , 39 ], sparse representation (SR)-based [ 40 , 41 , 42 , 43 , 44 ], graph-based [ 45 , 46 , 47 ], correlation filter-based [ 48 , 49 , 50 , 51 ], and deep learning-based approaches. Earlier studies used manual features to perform the appearance modeling of the target object.…”

Section: Related Workmentioning

confidence: 99%

Channel Exchanging for RGB-T Tracking

Zhao

Zhu

Ren

et al. 2021

Sensors

View full text Add to dashboard Cite

It is difficult to achieve all-weather visual object tracking in an open environment only utilizing single modality data input. Due to the complementarity of RGB and thermal infrared (TIR) data in various complex environments, a more robust object tracking framework can be obtained using video data of these two modalities. The fusion methods of RGB and TIR data are the core elements to determine the performance of the RGB-T object tracking method, and the existing RGB-T trackers have not solved this problem well. In order to solve the current low utilization of information intra single modality in aggregation-based methods and between two modalities in alignment-based methods, we used DiMP as the baseline tracker to design an RGB-T object tracking framework channel exchanging DiMP (CEDiMP) based on channel exchanging. CEDiMP achieves dynamic channel exchanging between sub-networks of different modes hardly adding any parameters during the feature fusion process. The expression ability of the deep features generated by our data fusion method based on channel exchanging is stronger. At the same time, in order to solve the poor generalization ability of the existing RGB-T object tracking methods and the poor ability in the long-term object tracking, more training of CEDiMP on the synthetic dataset LaSOT-RGBT is added. A large number of experiments demonstrate the effectiveness of the proposed model. CEDiMP achieves the best performance on two RGB-T object tracking benchmark datasets, GTOT and RGBT234, and performs outstandingly in the generalization testing.

show abstract

Thermal infrared and visible sequences fusion tracking based on a hybrid tracking framework with adaptive weighting scheme

Cited by 53 publications

References 8 publications

Exploring Fusion Strategies for Accurate RGBT Visual Object Tracking

Exploring Fusion Strategies for Accurate RGBT Visual Object Tracking

An Approach to Semantically Segmenting Building Components and Outdoor Scenes Based on Multichannel Aerial Imagery Datasets

Channel Exchanging for RGB-T Tracking

Contact Info

Product

Resources

About