Multi-domain collaborative feature representation for robust visual object tracking

Abstract

Jointly exploiting multiple different yet complementary domain information has been proven to be an effective way to perform robust object tracking. This paper focuses on effectively representing and utilizing complementary features from the frame domain and event domain for boosting object tracking performance in challenge scenarios. Specifically, we propose common features extractor to learn potential common representations from the RGB domain and event domain. For learning the unique features of the two domains, we utilize a unique extractor for event based on Spiking neural networks to extract edge cues in the event domain which may be missed in RGB in some challenging conditions, and a unique extractor for RGB based on deep convolutional neural networks to extract texture and semantic information in RGB domain. Extensive experiments on standard RGB benchmark and real event tracking dataset demonstrate the effectiveness of the proposed approach. We show our approach outperforms all compared state-of-the-art tracking algorithms and verify event-based data is a powerful cue for tracking in challenging scenes. Keywords Visual object tracking • Event-based camera • Multi-domain • Challenging conditions Jiqing Zhang and Kai Zhao have contributed equally to this study.

Visual Object Tracking Based on Deep Neural Network

“…In the testing phase, there is no need to fine-tune the network online, and the tracking is performed directly to achieve real-time tracking. Reference [24] proposes a novel multidomain convolutional neural network, which consists of multiple branches of shared layers and domain-specific layers. A model is pretrained by a large number of auxiliary images, and then the shared layer of the pretrained network is combined with the classification layer updated in real time, and finally the tracking is realized.…”

SCA-MMA: Spatial and Channel-Aware Multi-Modal Adaptation for Robust RGB-T Object Tracking

“…Jung et al [3] introduced the RoIAligh method to extract more accurate representations for the specific target. In [26,27], multi-domain feature representation networks have been proposed to perform information fusion across frame and event domains for improving the performance of the visual object tracking task. A semi-supervised multi-domain tracking framework [28] was proposed to learn the domain-invariant and domain-specific representations through employing an adversarial regularization.…”

Egomotion from event-based SNN optical flow

“…The combination of event cameras and SNNs has raised increasing interest in different areas of computer vision. They have mostly been used to tackle lowlevel vision tasks such as feature extraction [13], tracking [18,19,32], depth estimation [26], optical flow [12,15,23,24,33], motion segmentation [8,29], and image reconstruction [36], but also some higher-level vision tasks such as object or gesture detection and recognition [2,9,22]. Limited work has been done in using SNNs for camera egomotion estimation [11].…”