Multimodal End-to-End Autonomous Driving

Xiao, Yi; Codevilla, Felipe; Gurram, Akhil; Urfalıoḡlu, Onay; López, Antonio

doi:10.1109/tits.2020.3013234

Cited by 149 publications

(71 citation statements)

References 59 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…3, the textual representation (i.e., a word embedding) is very sparse when compared to the image one, which makes it very challenging to combine these two different representations into a unified model. As another example, when the driver of a car is driving autonomously, he probably has a LiDAR camera and other embedded sensors (e.g., depth sensors, etc) [81] to perceive his surroundings. Here, poor weather conditions can affect the visual perception of the environment.…”

Section: Data Acquisition and Samplingmentioning

confidence: 99%

A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets

et al. 2021

View full text Add to dashboard Cite

The research progress in multimodal learning has grown rapidly over the last decade in several areas, especially in computer vision. The growing potential of multimodal data streams and deep learning algorithms has contributed to the increasing universality of deep multimodal learning. This involves the development of models capable of processing and analyzing the multimodal information uniformly. Unstructured real-world data can inherently take many forms, also known as modalities, often including visual and textual content. Extracting relevant patterns from this kind of data is still a motivating goal for researchers in deep learning. In this paper, we seek to improve the understanding of key concepts and algorithms of deep multimodal learning for the computer vision community by exploring how to generate deep models that consider the integration and combination of heterogeneous visual cues across sensory modalities. In particular, we summarize six perspectives from the current literature on deep multimodal learning, namely: multimodal data representation, multimodal fusion (i.e., both traditional and deep learning-based schemes), multitask learning, multimodal alignment, multimodal transfer learning, and zero-shot learning. We also survey current multimodal applications and present a collection of benchmark datasets for solving problems in various vision domains. Finally, we highlight the limitations and challenges of deep multimodal learning and provide insights and directions for future research.

show abstract

Section: Data Acquisition and Samplingmentioning

confidence: 99%

A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets

et al. 2021

View full text Add to dashboard Cite

show abstract

“…In autonomous driving, end-to-end trained controllers learn from raw perception data, as opposed to maps [15] or other object representations [16]- [18]. Previous works have explored learning with expert information for lane following [1], [2], [19], [20], full point-to-point navigation [3], [8], [21], and shared human-robot control [22], [23], as well as in the context of RL by allowing the vehicle to repeatedly drive off the road [4]. However, when trained using state-of-the-art model-based simulation engines, these techniques are unable to be directly deployed in real-world driving conditions.…”

Section: Related Workmentioning

confidence: 99%

Learning Robust Control Policies for End-to-End Autonomous Driving From Data-Driven Simulation

Amini

Gilitschenski

Phillips

et al. 2020

IEEE Robot. Autom. Lett.

160

View full text Add to dashboard Cite

In this work, we present a data-driven simulation and training engine capable of learning end-to-end autonomous vehicle control policies using only sparse rewards. By leveraging real, human-collected trajectories through an environment, we render novel training data that allows virtual agents to drive along a continuum of new local trajectories consistent with the road appearance and semantics, each with a different view of the scene. We demonstrate the ability of policies learned within our simulator to generalize to and navigate in previously unseen real-world roads, without access to any human control labels during training. Our results validate the learned policy onboard a full-scale autonomous vehicle, including in previously un-encountered scenarios, such as new roads and novel, complex, near-crash situations. Our methods are scalable, leverage reinforcement learning, and apply broadly to situations requiring effective perception and robust operation in the physical world.Index Terms-Deep learning in robotics and automation, autonomous agents, real world reinforcement learning, data-driven simulation. I. INTRODUCTIONE ND-TO-END (i.e., perception-to-control) trained neural networks for autonomous vehicles have shown great promise for lane stable driving [1]-[3]. However, they lack methods to learn robust models at scale and require vast amounts of training data that are time consuming and expensive to collect. Learned end-to-end driving policies and modular perception components in a driving pipeline require capturing training data from all necessary edge cases, such as recovery from off-orientation positions or even near collisions. This is not only prohibitively expensive, but also potentially dangerous [4]. Training and evaluating robotic controllers in simulation [5]-[7]

show abstract

“…However, they used the World Rally Championship 6 (WRC6) environment for their experiments, which is not for research use and not open source; thus, it is unsuitable to use it for benchmark generalization. Using multimodal inputs can improve end-to-end driving [23]. They trained their end-to-end driving agent with RGB images, depth images and measurements.…”

Section: Background a End-to-end Drivingmentioning

confidence: 99%

A New Open-Source Off-Road Environment for Benchmark Generalization of Autonomous Driving

Han¹,

Park

Kim

2021

IEEE Access

View full text Add to dashboard Cite

Recently, deep neural networks have greatly improved autonomous driving. However, as a great deal of training data is required, most studies have employed simulators. Generalization of such driving is key in terms of safety. The simulated environments feature only small variations in favorable conditions and thus cannot be used for benchmarking. Therefore, we developed a new open-source (OpenAI Gym-like) off-road environment featuring differently structured forests, plateaus, deserts, and snowfields. The dynamic topographical structures make the off-road environment a very challenging generalization problem. Our offroad environment can precisely evaluate autonomous driving in terms of generalization. Additionally, we proposed an evaluation method based on the success rate of driving tasks, enabling effective driving ability measurement. Furthermore, we evaluate the performance of existing end-to-end driving methods in our off-road environment. The results show that the end-to-end driving methods lack generalization ability and fail to generalize to unseen environments. Our off-road environment can help autonomous driving researchers develop a better, generalizable driving system. Unreal engine-level assets and codes are available at github.com a . We briefly introduce our model in https://www.youtube.com/watch?v=YIKd5bcHcGA.

show abstract

Multimodal End-to-End Autonomous Driving

Cited by 149 publications

References 59 publications

A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets

A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets

Learning Robust Control Policies for End-to-End Autonomous Driving From Data-Driven Simulation

A New Open-Source Off-Road Environment for Benchmark Generalization of Autonomous Driving

Contact Info

Product

Resources

About