MotionNet: Joint Perception and Motion Prediction for Autonomous Driving Based on Bird’s Eye View Maps

Wu, Pengxiang; Chen, Siheng; Metaxas, Dimitris N.

doi:10.1109/cvpr42600.2020.01140

Cited by 135 publications

(135 citation statements)

References 40 publications

Supporting

Mentioning

135

Contrasting

Order By: Relevance

“…Such complete features enable the backbone network to make predictions with high confidence, allowing the outcomes to be safely used in later pipelines like planning and control. Conducted experiments prove the superiority of LiCaNet over our previous work [12], MotionNet and thus other state-of-the-art models reported in [14]. LiCaNet achieves outstanding accuracy in real-time for perception and motion prediction, especially for small and distant objects.…”

Section: Introductionmentioning

confidence: 52%

“…As LIDARs are the most common sensor used in autonomous driving, we discuss different ways available in the literature to represent LIDAR data. Point clouds can be processed in point-based form [9], 3D voxelization [10], [11], bird's-eye view (BEV) [12], [13], [3], [14], and in range view (RV) [12], [15], [16], [3], [17], [18]. Processing 3D LIDAR point cloud in its raw form is straightforward and requires no transformation.…”

Section: Introductionmentioning

confidence: 99%

“…Thus, LiCaNet involves combining historical LIDAR data encoded in BEV form with RV representation, along with a camera image. Our generated multimodal features are then fed into MotionNet backbone [14] to perform accurate joint perception and motion predictions. MotionNet is a real-time model that perceives and predicts motion at a pixel level with state-of-the-art results.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

LiCaNet: Further Enhancement of Joint Perception and Motion Prediction based on Multi-Modal Fusion

Khalil¹,

Mouftah²

2021

Preprint

View full text Add to dashboard Cite

<p>The safety and reliability of autonomous driving pivots on the accuracy of perception and motion prediction pipelines, which in turn reckons primarily on the sensors deployed onboard. Slight confusion in perception and motion prediction can result in catastrophic consequences due to misinterpretation in later pipelines. Therefore, researchers have recently devoted considerable effort towards developing accurate perception and motion prediction models. To that end, we propose LIDAR Camera network (LiCaNet) that leverages multi-modal fusion to further enhance the joint perception and motion prediction performance accomplished in our earlier work. LiCaNet expands on our previous fusion network by adding a camera image to the fusion of RV image with historical BEV data sourced from a LIDAR sensor. We present a comprehensive evaluation to validate the outstanding performance of LiCaNet compared to the state-of-the-art. Experiments reveal that utilizing a camera sensor results in a substantial perception gain over our previous fusion network and a steep reduction in displacement errors. Moreover, the majority of the achieved improvement falls within camera range, with the highest registered for small and distant objects, confirming the significance of incorporating a camera sensor into a fusion network.</p>

show abstract

Section: Introductionmentioning

confidence: 52%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

LiCaNet: Further Enhancement of Joint Perception and Motion Prediction based on Multi-Modal Fusion

Khalil¹,

Mouftah²

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Models and Dataset. For our evaluation, we use the models detailed in Section 3.3, with PointPillars [8] and SECOND [14] used for 3D Object Detection and MotionNet [12] for the Object-motion Prediction. For AV driving scenes, we use the mini dataset from nuScenes [1], which contains 3D point-clouds of 10 driving scenes.…”

Section: Experiments and Resultsmentioning

confidence: 99%

Temporal Consistency Checks to Detect LiDAR Spoofing Attacks on Autonomous Vehicle Perception

You

Hau

Demetriou

2021

Proceedings of the 1st Workshop on Security and Privacy for Mobile AI

View full text Add to dashboard Cite

LiDAR sensors are used widely in Autonomous Vehicles for better perceiving the environment which enables safer driving decisions. Recent work has demonstrated serious LiDAR spoofing attacks with alarming consequences. In particular, model-level LiDAR spoofing attacks aim to inject fake depth measurements to elicit ghost objects that are erroneously detected by 3D Object Detectors, resulting in hazardous driving decisions. In this work, we explore the use of motion as a physical invariant of genuine objects for detecting such attacks. Based on this, we propose a general methodology, 3D Temporal Consistency Check (3D-TC2), which leverages spatiotemporal information from motion prediction to verify objects detected by 3D Object Detectors. Our preliminary design and implementation of a 3D-TC2 prototype demonstrates very promising performance, providing more than 98% attack detection rate with a recall of 91% for detecting spoofed Vehicle (Car) objects, and is able to achieve real-time detection at 41Hz. CCS CONCEPTS• Security and privacy → Intrusion/anomaly detection and malware mitigation; • Computing methodologies → Computer vision; Scene anomaly detection.

show abstract

“…[16] proposes a method called LiDAR-flow, which provides robust estimation of dense scene flows by fusing sparse LiDAR with stereo images. [17] and [20] fusing temporal information to detect dynamic objects, they use multi frame point cloud as input to regress the motion behavior of objects on the aerial view through the network, the advantage of this kind of method is to detect all moving objects in the field of vision of LiDAR, including objects not seen in the training set, which is of great significance to the safety of robotics automatic driving.…”

Section: B Lidar Dynamic Object Detectionmentioning

confidence: 99%

DLOAM: Real-time and Robust LiDAR SLAM System Based on CNN in Dynamic Urban Environments

Liu

Sun

Liu

2024

IEEE Open J. Intell. Transp. Syst.

View full text Add to dashboard Cite

Dynamic object detection, state estimation, and map-building are crucial for autonomous robot systems and intelligent transportation applications in urban scenarios. Most current LiDAR Simultaneous Localization and Mapping (SLAM) systems operate on the assumption that the observed environment is static. However, the overall accuracy and robustness of a SLAM system can be compromised by dynamic objects in the environment. Aiming at the problem of inaccurate odometry estimation and wrong mapping caused by the existing LiDAR SLAM method which cannot detect the dynamic objects, we study the SLAM problem of robots and unmanned vehicles equipped with LiDAR traveling in the dynamic urban scenes. We propose a fast LiDAR-only model-free dynamic objects detection method, which uses the spatial and temporal information of point cloud through a convolutional neural network (CNN), and the detection accuracy is improved by 35% to 86% compared with methods that only use spatial information. We further integrate it into a state-of-the-art LiDAR SLAM framework to improve the SLAM performance. Firstly, the range image constructed by LiDAR point cloud is used for ground extraction and non-ground point clustering. Then, the motion of objects in the scene is estimated by the difference between adjacent frames, and the segmented objects are further divided into dynamic objects and static objects by their motion features. After that, the stable feature points are extracted from the static objects. Finally, the pose transformation of adjacent frames is solved by matching feature point pairs. We evaluated the accuracy and robustness of our system on datasets with different challenging dynamic environments, and the results show our system has significant improvements in accuracy and robustness of odometry and mapping, while still maintain real-time performance, which is sufficient for autonomous robot systems and intelligent transportation applications in urban scenarios.

show abstract

MotionNet: Joint Perception and Motion Prediction for Autonomous Driving Based on Bird’s Eye View Maps

Cited by 135 publications

References 40 publications

LiCaNet: Further Enhancement of Joint Perception and Motion Prediction based on Multi-Modal Fusion

LiCaNet: Further Enhancement of Joint Perception and Motion Prediction based on Multi-Modal Fusion

Temporal Consistency Checks to Detect LiDAR Spoofing Attacks on Autonomous Vehicle Perception

DLOAM: Real-time and Robust LiDAR SLAM System Based on CNN in Dynamic Urban Environments

Contact Info

Product

Resources

About