RGBD1K: A Large-Scale Dataset and Benchmark for RGB-D Object Tracking

Abstract

RGB-D object tracking has attracted considerable attention recently, achieving promising performance thanks to the symbiosis between visual and depth channels. However, given a limited amount of annotated RGB-D tracking data, most state-of-the-art RGB-D trackers are simple extensions of high-performance RGB-only trackers, without fully exploiting the underlying potential of the depth channel in the offline training stage. To address the dataset deficiency issue, a new RGB-D dataset named RGBD1K is released in this paper. The RGBD1K contains 1,050 sequences with about 2.5M frames in total. To demonstrate the benefits of training on a larger RGB-D data set in general, and RGBD1K in particular, we develop a transformer-based RGB-D tracker, named SPT, as a baseline for future visual object tracking studies using the new dataset. The results, of extensive experiments using the SPT tracker demonstrate the potential of the RGBD1K dataset to improve the performance of RGB-D tracking, inspiring future developments of effective tracker designs. The dataset and codes will be available on the project homepage: https://github.com/xuefeng-zhu5/RGBD1K.

Generative-Based Fusion Mechanism for Multi-Modal Tracking

“…The effectiveness of GMMT is verified on GTOT (Li et al 2016), LasHeR (Li et al 2022), and RGBD1K (Zhu et al 2023b) benchmarks. In these benchmarks, precision rate (PR), success rate (SR), normalised precision rate (NPR), recall (RE), and F-score are employed for evaluation, whose detail introductions can be found in the supplementary material.…”

“…ViPT-D is an extension of ViPT tailored for RGB-D data.. Initially, we run the official ViPT-D on the RGBD1K dataset, but we notice a performance gap compared to the state-of-the-art, SPT (Zhu et al 2023b). As RGBD1K videos exhibit a higher diversity with more challenging factors compared to other RGB-D benchmarks, we retrain ViPT-D on the training split of RGBD1K.…”

“…Due to the strict demand for the robustness of tracking systems in real-world applications, such as surveillance (Lu et al 2023) and unmanned driving (Zhang et al 2023a), visual object tracking with an auxiliary modality, named as multi-modal tracking, draws growing attention recently. For example, the thermal infrared (TIR) modality provides more stable scene perception in the nighttime (Tang et al 2023), and the depth (D) modality provides 3-D perception against occlusions (Zhu et al 2023b). In other words, the use of auxiliary modalities can complement the visible image in challenging scenarios.…”

“…In order to learn the external projection between the input and output, as well as the internal data distributions, GMs require a longer training time and a bigger size of training data. However, despite the emergence of several multi-modal datasets in recent years (Zhu et al 2023b;Li et al 2022;Zhang et al 2022a), containing around 1000 videos captured across less than 1000 scenarios, there remains a significant diversity gap compared to some widely-used datasets for generation tasks, such as CelebA (Liu et al 2015) used in face generation. Therefore, the multi-modal feature pair grouped with a random factor is formulated as the input of the GMMT to enlarge the data size and avoid over-fitting.…”

“…Consistent improvements can be obtained on all the evaluation metrics. Furthermore, extended experiments on the largest RGB-D benchmark (Zhu et al 2023b) are also conducted to demonstrate the generalisation of GMMT. In conclusion, our contributions can be summarised as follows:…”