Multi-Modal Fusion Transformer for End-to-End Autonomous Driving

Abstract

How should representations from complementary sensors be integrated for autonomous driving? Geometrybased sensor fusion has shown great promise for perception tasks such as object detection and motion forecasting. However, for the actual driving task, the global context of the 3D scene is key, e.g. a change in traffic light state can affect the behavior of a vehicle geometrically distant from that traffic light. Geometry alone may therefore be insufficient for effectively fusing representations in end-to-end driving models. In this work, we demonstrate that imitation learning policies based on existing sensor fusion methods under-perform in the presence of a high density of dynamic agents and complex scenarios, which require global contextual reasoning, such as handling traffic oncoming from multiple directions at uncontrolled intersections. Therefore, we propose TransFuser, a novel Multi-Modal Fusion Transformer, to integrate image and LiDAR representations using attention. We experimentally validate the efficacy of our approach in urban settings involving complex scenarios using the CARLA urban driving simulator. Our approach achieves state-of-the-art driving performance while reducing collisions by 76% compared to geometry-based fusion.

Improving Multi-Modal Learning with Uni-Modal Teachers

“…Most state-of-the-art methods [23,34,41,19,25] for RGB-Depth semantic segmentation leveraged an RGB encoder and a depth encoder respectively, and then applied middle fusion between them to better incorporate low-level (texture, geometry) and high-level (semantic) features. In the field of autonomous driving, there have also been great efforts on fusing camera frames and LiDAR scans for better object detection [29,28,37].…”

An Anti-Noise Gearbox Fault Diagnosis Method based on Multi-Scale Transformer Convolution and Transfer Learning

“…To be specific, all the related operations in the Transformer network are order-independent and parallelizable. With the rapid development of the Transformer network, it has been widely applied and demonstrated its outstanding performance in many fields, such as image processing, [13], pose recognition, [14], autonomous driving, [15], and natural language processing, [16], [17]. Moreover, owing to the interaction and coupling effects among different components and subsystems of the gearbox, the measured vibration signals collected from sensors installed on the house usually contain multiple intrinsic oscillatory modes, [18].…”