3D-SiamRPN: An End-to-End Learning Method for Real-Time 3D Single Object Tracking Using Raw Point Cloud

Abstract

3D single object tracking is a key issue for autonomous following robot, where the robot should robustly track and accurately localize the target for efficient following. In this paper, we propose a 3D tracking method called 3D-SiamRPN Network to track a single target object by using raw 3D point cloud data. The proposed network consists of two subnetworks. The first subnetwork is feature embedding subnetwork which is used for point cloud feature extraction and fusion. In this subnetwork, we first use PointNet++ to extract features of point cloud from template and search branches. Then, to fuse the information of features in the two branches and obtain their similarity, we propose two cross correlation modules, named Pointcloud-wise and Point-wise respectively. The second subnetwork is region proposal network(RPN), which is used to get the final 3D bounding box of the target object based on the fusion feature from cross correlation modules. In this subnetwork, we utilize the regression and classification branches of a region proposal subnetwork to obtain proposals and scores, thus get the final 3D bounding box of the target object. Experimental results on KITTI dataset show that our method has a competitive performance in both Success and Precision compared to the state-of-the-art methods, and could run in real-time at 20.8 FPS. Additionally, experimental results on H3D dataset demonstrate that our method also has good generalization ability and could achieve good tracking performance in a new scene without re-training. Index Terms-3D single object tracking, LIDAR point-cloud, siamese network, region proposal network. I. INTRODUCTIONO BJECT tracking, which is a key issue in computer vision and robotics, could be used in a wide range of applications, such as augmented reality, self-driving cars and mobile robotics. Traditionally, cameras are widely used for object tracking, known as visual object tracking (VOT). For mobile robots or self-driving cars, 3D object tracking is usually more Manuscript

Deep Learning Inspired Object Consolidation Approaches Using LiDAR Data for Autonomous Driving: A Review

“…3D single object tracking is still a challenge. In this regard, the 3D-SiamRPN method has been designed to formalize the 3D single object tracking issue [118]. The author considered PointNet++ to estimate the features, and they designed a Point-wise cross-correlation model for effective data fusion.…”

Online Adaptation for Implicit Object Tracking and Shape Reconstruction in the Wild

“…The second paradigm is model-free. It leverages the tracking results in previous frames and their similarities to the current frame for tracking [19,22,52,36,14]. For example, Giancola et al [19] first proposed to apply a Siamese network in 3D for matching and exploit a shape completion network for regularization.…”

“…We further evaluate our method on the KITTI dataset. We compare our method with SC3D [19], P2B [36], 3D-SiamRPN [15] and BAT [54], which are the recent single object tracking methods on the KITTI dataset. Since all methods employ the annotated bounding box for training while our method only pretrains on ShapeNet, we employ the detection term defined in Eq.…”

PTTR: Relational 3D Point Cloud Object Tracking with Transformer

“…SA-P2B [39] proposes to learn the object structure as an auxiliary task. 3D-SiamRPN [7] uses a RPN [22] head to predict the final results. All existing SOT methods either use cosine similarity or cross-correlation to match the search and template features, which are essentially linear matching processes and cannot adapt to complex situations where random noise and occlusions are involved.…”

“…Following P2B, SA-P2B [39] adds an extra auxiliary network to predict the object structure. In a similar framework, 3D-SiamRPN [7] uses a cross-correlation module for feature matching and an RPN head for final prediction. These methods [9,21,39] essentially perform a linear matching process between features in the search domain and the template, which cannot adapt to different 3D observations caused by random noise, sparsity, and occlusions.…”