Multi-View Fusion of Sensor Data for Improved Perception and Prediction in Autonomous Driving

Fadadu, Sudeep; Pandey, Shreyash; Hegde, Darshan; Shi, Yi; Chou, Fang‐Chieh; Djuric, Nemanja; Vallespi-Gonzalez, Carlos

doi:10.1109/wacv51458.2022.00335

Cited by 38 publications

(15 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Despite the fact that multi-modal sensor configurations are widely seen on self-driving vehicles [38,37,55,22,4,12], research on multi-modal sensor attacks is still very limited. Several preliminary works show the possibility of attacking multi-sensor fusion networks [61,7,72].…”

Section: Related Workmentioning

confidence: 99%

“…We focus on attacking the LiDAR-image object detector MMF [37], a state-of-the-art multi-sensor network architecture employed in modern self-driving systems. We believe our study is general enough as the multi-sensor fusion module is a common building block for other related works [22,4,12,69]. Specifically, we require the adversarial multi-sensory attack to be (1) input-agnostic so that it can be applied in different environments, (2) geometricallyconsistent across image and LiDAR input modalities, and (3) fully-automatic for implementation at large-scale.…”

Section: Multi-sensor Adversarial Learningmentioning

confidence: 99%

See 1 more Smart Citation

Exploring Adversarial Robustness of Multi-Sensor Perception Systems in Self Driving

Tu¹,

Li²,

Yan³

et al. 2021

Preprint

View full text Add to dashboard Cite

Modern self-driving perception systems have been shown to improve upon processing complementary inputs such as LiDAR with images. In isolation, 2D images have been found to be extremely vulnerable to adversarial attacks. Yet, there have been limited studies on the adversarial robustness of multi-modal models that fuse LiDAR features with image features. Furthermore, existing works do not consider physically realizable perturbations that are consistent across the input modalities. In this paper, we showcase practical susceptibilities of multi-sensor detection by placing an adversarial object on top of a host vehicle. We focus on physically realizable and input-agnostic attacks as they are feasible to execute in practice, and show that a single universal adversary can hide different host vehicles from state-of-the-art multi-modal detectors. Our experiments demonstrate that successful attacks are primarily caused by easily corrupted image features. Furthermore, we find that in modern sensor fusion methods which project image features into 3D, adversarial attacks can exploit the projection process to generate false positives across distant regions in 3D. Towards more robust multi-modal perception systems, we show that adversarial training with feature denoising can boost robustness to such attacks significantly. However, we find that standard adversarial defenses still struggle to prevent false positives which are also caused by inaccurate associations between 3D LiDAR points and 2D pixels.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Multi-sensor Adversarial Learningmentioning

confidence: 99%

Exploring Adversarial Robustness of Multi-Sensor Perception Systems in Self Driving

Tu¹,

Li²,

Yan³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Sensor Fusion Methods for Object Detection and Motion Forecasting: The majority of the sensor fusion works consider perception tasks, e.g. object detection [22,12,66,7,44,31,34,61,33,37] and motion forecasting [36,5,35,63,6,19,38]. They operate on multi-view LiDAR, e.g.…”

Section: Related Workmentioning

confidence: 99%

“…Prior works in the field of sensor fusion have mostly focused on the perception aspect of driving, e.g. 2D and 3D object detection [22,12,66,9,44,31,34,61,33,37], motion forecasting [22,36,5,35,63,6,19,38,32,9], and depth estimation [24,60,61,33]. These methods focus on learning a state representation that captures the geometric and semantic information of the 3D scene.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Multi-Modal Fusion Transformer for End-to-End Autonomous Driving

Prakash¹,

Chitta²,

Geiger³

2021

Preprint

View full text Add to dashboard Cite

How should representations from complementary sensors be integrated for autonomous driving? Geometrybased sensor fusion has shown great promise for perception tasks such as object detection and motion forecasting. However, for the actual driving task, the global context of the 3D scene is key, e.g. a change in traffic light state can affect the behavior of a vehicle geometrically distant from that traffic light. Geometry alone may therefore be insufficient for effectively fusing representations in end-to-end driving models. In this work, we demonstrate that imitation learning policies based on existing sensor fusion methods under-perform in the presence of a high density of dynamic agents and complex scenarios, which require global contextual reasoning, such as handling traffic oncoming from multiple directions at uncontrolled intersections. Therefore, we propose TransFuser, a novel Multi-Modal Fusion Transformer, to integrate image and LiDAR representations using attention. We experimentally validate the efficacy of our approach in urban settings involving complex scenarios using the CARLA urban driving simulator. Our approach achieves state-of-the-art driving performance while reducing collisions by 76% compared to geometry-based fusion.

show abstract

Semantic Synthesis of Pedestrian Locomotion

Priisalu

Păduraru

Pirinen

et al. 2021

Computer Vision – ACCV 2020

View full text Add to dashboard Cite

It is difficult to perform 3D reconstruction from on-vehicle gathered video due to the large forward motion of the vehicle. Even object detection and human sensing models perform significantly worse on onboard videos when compared to standard benchmarks because objects often appear far away from the camera compared to the standard object detection benchmarks, image quality is often decreased by motion blur and occlusions occur often. This has led to the popularisation of traffic data-specific benchmarks. Recently Light Detection And Ranging (LiDAR) sensors have become popular to directly estimate depths without the need to perform 3D reconstructions. However, LiDAR-based methods still lack in articulated human detection at a distance when compared to image-based methods. We hypothesize that benchmarks targeted at articulated human sensing from LiDAR data could bring about increased research in human sensing and prediction in traffic and could lead to improved traffic safety for pedestrians.

show abstract

Multi-View Fusion of Sensor Data for Improved Perception and Prediction in Autonomous Driving

Cited by 38 publications

References 25 publications

Exploring Adversarial Robustness of Multi-Sensor Perception Systems in Self Driving

Exploring Adversarial Robustness of Multi-Sensor Perception Systems in Self Driving

Multi-Modal Fusion Transformer for End-to-End Autonomous Driving

Semantic Synthesis of Pedestrian Locomotion

Contact Info

Product

Resources

About