Multi-View Fusion of Sensor Data for Improved Perception and Prediction in Autonomous Driving

Abstract

We present an end-to-end method for object detection and trajectory prediction utilizing multi-view representations of LiDAR returns. Our method builds on a state-of-the-art Bird's-Eye View (BEV) network that fuses voxelized features from a sequence of historical LiDAR data as well as rasterized highdefinition map to perform detection and prediction tasks. We extend the BEV network with additional LiDAR Range-View (RV) features that use the raw LiDAR information in its native, non-quantized representation. The RV feature map is projected into BEV and fused with the BEV features computed from LiDAR and high-definition map. The fused features are then further processed to output the final detections and trajectories, within a single end-to-end trainable network. In addition, using this framework the RV fusion of LiDAR and camera is performed in a straightforward and computational efficient manner. The proposed approach improves the state-of-the-art on proprietary large-scale real-world data collected by a fleet of self-driving vehicles, as well as on the public nuScenes data set.

MVFuseNet: Improving End-to-End Object Detection and Motion Forecasting through Multi-View Fusion of LiDAR Data

“…Notably, our method shows a ∼ 15% improvement on pedestrian detection, a ∼ 40% improvement on bike detection, and a ∼ 30% improvement on motion forecasting of bikes, as compared to the best BEV-only MultiXNet. Next, we compare our method to another recent multi-view method L-MV [10]. As shown in Table 1, our method outperforms L-MV [10] on all classes by a large margin on both detection and forecasting.…”

“…Recently, [10] proposed a multi-view approach for the joint task. In this method, the authors proposed fusing a single-frame RV projection with multiple frames of BEV projection, which improves object detection performance.…”

“…These methods show that recent work on multi-modal predictions and the use of interaction graphs to model complex relationships can be easily extended to the framework of joint object detection and motion forecasting. [5] and [10] are recent multi-sensor methods that build on top of [4] by using radar and camera inputs respectively. These methods, by virtue of operating in BEV, lose out on high-resolution point information and are often limited by range of operation.…”

“…In this section, we compare our method to existing endto-end methods using the evaluation setting of [4,10]. As shown in Table 1, our novel multi-view method significantly outperforms all other methods, on both detection and forecasting tasks for all evaluated classes.…”

“…This provides an edge to existing BEV methods on motion forecasting tasks; however, their scalability to operate in large areas remains a challenge. There has been some recent work on using multiple views for detection [9,10], but the space of models that can efficiently use multiple views for end-toend detection and motion forecasting remains largely unexplored.…”