RGBT Tracking via Multi-Adapter Network with Hierarchical Divergence Loss

Lu, Andong; Li, Chenglong; Yan, Yuqing; Tang, Jin; Luo, Bin

doi:10.1109/tip.2021.3087341

Cited by 96 publications

(26 citation statements)

References 38 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We test our Attribute-Based Progressive Fusion Network (APFNet) on three popular RGBT tracking benchmarks and compare performance with some state-of-the-art trackers, such as ADRNet (Zhang et al 2021a), MANet++ (Lu et al 2021), M 5 L (Tu et al 2021), CMPP (Wang et al 2020), JM-MAC (Zhang et al 2021b), CAT (Li et al 2020), MANet (Li et al 2019b), DAFNet (Gao et al 2019), mfDiMP (Zhang et al 2019), DAPNet (Zhu et al 2019), FANet (Zhu et al 2020), MaCNet (Zhang et al 2020) and MDNet (Nam and Han 2016), to validate the effectiveness of proposed method.…”

Section: Quantitative Comparisonmentioning

confidence: 99%

Attribute-Based Progressive Fusion Network for RGBT Tracking

Xiao

Yang

et al. 2022

AAAI

Self Cite

View full text Add to dashboard Cite

RGBT tracking usually suffers from various challenge factors, such as fast motion, scale variation, illumination variation, thermal crossover and occlusion, to name a few. Existing works often study fusion models to solve all challenges simultaneously, and it requires fusion models complex enough and training data large enough, which are usually difficult to be constructed in real-world scenarios. In this work, we disentangle the fusion process via the challenge attributes, and thus propose a novel Attribute-based Progressive Fusion Network (APFNet) to increase the fusion capacity with a small number of parameters while reducing the dependence on large-scale training data. In particular, we design five attribute-specific fusion branches to integrate RGB and thermal features under the challenges of thermal crossover, illumination variation, scale variation, occlusion and fast motion respectively. By disentangling the fusion process, we can use a small number of parameters for each branch to achieve robust fusion of different modalities and train each branch using the small training subset with the corresponding attribute annotation. Then, to adaptive fuse features of all branches, we design an aggregation fusion module based on SKNet. Finally, we also design an enhancement fusion transformer to strengthen the aggregated feature and modality-specific features. Experimental results on benchmark datasets demonstrate the effectiveness of our APFNet against other state-of-the-art methods.

show abstract

Section: Quantitative Comparisonmentioning

confidence: 99%

Attribute-Based Progressive Fusion Network for RGBT Tracking

Xiao

Yang

et al. 2022

AAAI

Self Cite

View full text Add to dashboard Cite

show abstract

“…For example, Zhang et al [32] proposed a pixel-level fusion-based RGB-T tracker. In contrast, some RGB-T trackers are based on feature-level fusion [7,33,34] or decision-level [35] or combine several fusion levels [9]. The performance of RGB-T trackers have been significantly improved.…”

Section: B Rgb-t Trackingmentioning

confidence: 99%

“…To improve tracking performance, researchers have used thermal images and RGB images together to perform RGB-T tracking [1][2][3][4][5][6][7]. This is based on the fact that thermal images are insensitive to illumination changes while RGB images contain more texture details [8].…”

Section: Introductionmentioning

confidence: 99%

Self-Supervised RGB-T Tracking with Cross-Input Consistency

Zhang¹,

Demiris²

2023

Preprint

View full text Add to dashboard Cite

In this paper, we propose a self-supervised RGB-T tracking method. Different from existing deep RGB-T trackers that are using a large number of annotated RGB-T image pairs for training, our RGB-T tracker is trained using unlabeled RGB-T video pairs in a self-supervised manner. We propose a novel cross-input consistency-based self-supervised training strategy based on the idea that tracking can be performed using different inputs. Specifically, we construct two distinct inputs using unlabeled RGB-T video pairs. We then track objects using these two inputs to generate results, based on which we construct our cross-input consistency loss. Meanwhile, we propose a reweighting strategy to make our loss function robust to low-quality training samples. We build our tracker on a Siamese correlation filter network. To the best of our knowledge, our tracker is the first self-supervised RGB-T tracker. Extensive experiments on two public RGB-T tracking benchmarks demonstrate that the proposed training strategy is effective. Remarkably, despite training only with a corpus of unlabeled RGB-T video pairs, our tracker outperforms seven supervised RGB-T trackers on the GTOT dataset.

show abstract

“…Then, the enhanced features were obtained by using cross-modal residual connections, and finally, these features were concatenated. Lu et al [ 35 ] designed an instance adapter to use two fully connected layers for each modal, and then predicted the modal weight to realize the quality-aware fusion of different modals.…”

Section: Related Workmentioning

confidence: 99%

Optical Flow-Aware-Based Multi-Modal Fusion Network for Violence Detection

Xiao

Gao

Wang

et al. 2022

Entropy

View full text Add to dashboard Cite

Violence detection aims to locate violent content in video frames. Improving the accuracy of violence detection is of great importance for security. However, the current methods do not make full use of the multi-modal vision and audio information, which affects the accuracy of violence detection. We found that the violence detection accuracy of different kinds of videos is related to the change of optical flow. With this in mind, we propose an optical flow-aware-based multi-modal fusion network (OAMFN) for violence detection. Specifically, we use three different fusion strategies to fully integrate multi-modal features. First, the main branch concatenates RGB features and audio features and the optical flow branch concatenates optical flow features with RGB features and audio features, respectively. Then, the cross-modal information fusion module integrates the features of different combinations and applies weights to them to capture cross-modal information in audio and video. After that, the channel attention module extracts valuable information by weighting the integration features. Furthermore, an optical flow-aware-based score fusion strategy is introduced to fuse features of different modalities from two branches. Compared with methods on the XD-Violence dataset, our multi-modal fusion network yields APs that are 83.09% and 1.4% higher than those of the state-of-the-art methods in offline detection, and 78.09% and 4.42% higher than those of the state-of-the-art methods in online detection.

show abstract

RGBT Tracking via Multi-Adapter Network with Hierarchical Divergence Loss

Cited by 96 publications

References 38 publications

Attribute-Based Progressive Fusion Network for RGBT Tracking

Attribute-Based Progressive Fusion Network for RGBT Tracking

Self-Supervised RGB-T Tracking with Cross-Input Consistency

Optical Flow-Aware-Based Multi-Modal Fusion Network for Violence Detection

Contact Info

Product

Resources

About