3D Object Tracking with Transformer

Abstract

Feature fusion and similarity computation are two core problems in 3D object tracking, especially for object tracking using sparse and disordered point clouds. Feature fusion could make similarity computing more efficient by including target object information. However, most existing LiDAR-based approaches directly use the extracted point cloud feature to compute similarity while ignoring the attention changes of object regions during tracking. In this paper, we propose a feature fusion network based on transformer architecture. Benefiting from the self-attention mechanism, the transformer encoder captures the inter-and intra-relations among different regions of the point cloud. By using cross-attention, the transformer decoder fuses features and includes more target cues into the current point cloud feature to compute the region attentions, which makes the similarity computing more efficient. Based on this feature fusion network, we propose an end-to-end point cloud object tracking framework, a simple yet effective method for 3D object tracking using point clouds. Comprehensive experimental results on the KITTI dataset show that our method achieves new state-of-the-art performance. Code is available at: https://github.com/3bobo/lttr.Recently, LiDAR-based 3D object tracking has been received more and more attention. Benefiting from the development of visual tracking [1,7,13,15,16], most 3D tracking methods [11,20,30] also use the Siamese-like tracking pipeline. The pipeline first inputs template point clouds of the target object and search point clouds of the current frame to its top and bottom branches respectively, then fuses the two-branch features based on similarity. Finally, the fused features are used to localize the position of the object to be tracked. However, compared with visual tracking, LiDAR-based tracking has more challenges due to the sparsity and disorder of the point clouds. For example, the point clouds will become much sparser with the increasing distance of the object, which hinders the feature extraction. Meanwhile,

CXTrack: Improving 3D Point Cloud Tracking with Contextual Information

“…V2B [9] proposes a voxel-to-BEV (Bird's Eye View) target localization network, which projects the point features into a dense BEV feature map to tackle the sparsity of point clouds. Inspired by the success of transformers in 2D vision tasks [2,4], LTTR [3] adopts a transformerbased architecture to fuse features from two branches and propagate target cues. PTT [24] integrates a transformer module into the P2B architecture to refine point features.…”

“…We make comprehensive comparisons on the KITTI dataset with previous state-of-the-art methods, including SC3D [7], P2B [22], 3DSiamRPN [5], LTTR [3], MLVS-Net [31], BAT [34], PTT [24], V2B [9], PTTR [36], STNet [10] and M2-Track [35]. As illustrated in Tab.…”

“…P2B reaches a balance between performance and speed. Therefore many follow-up works adopt the same paradigm [3,9,10,24,31,34,36]. However, both SC3D and P2D paradigms overlook the contextual information across two consecutive frames and rely entirely on the appearance of the target.…”

Beyond 3D Siamese Tracking: A Motion-Centric Paradigm for 3D Single Object Tracking in Point Clouds

“…Results on KITTI. We compare M 2 -Track with seven topperformance approaches [6,8,10,13,23,26,42,43], which have published results on KITTI. As shown in Tab.…”

“…Our method perfectly keeps track of the target while BAT almost fails. To handle such fastmoving objects, [6,42] leverage BEV-based RPN to generate high-recall proposals from a larger search region. In contrast, we handle this simply by motion modeling without sophisticated architectures.…”