End-to-End Model-Free Reinforcement Learning for Urban Driving Using Implicit Affordances

Abstract

Reinforcement Learning (RL) aims at learning an optimal behavior policy from its own experiments and not rulebased control methods. However, there is no RL algorithm yet capable of handling a task as difficult as urban driving. We present a novel technique, coined implicit affordances, to effectively leverage RL for urban driving thus including lane keeping, pedestrians and vehicles avoidance, and traffic light detection. To our knowledge we are the first to present a successful RL agent handling such a complex task especially regarding the traffic light detection. Furthermore, we have demonstrated the effectiveness of our method by winning the Camera Only track of the CARLA challenge.

Using Semantic Information to Improve Generalization of Reinforcement Learning Policies for Autonomous Driving

“…Recent work [30] showed significant improvement on autonomous driving with reinforcement learning, training a network with affordances (like semantic segmentation, or traffic light state), and then use these affordances to train a reinforcement learning agent. An ablation study is made to compare the performance on unseen town when using one or several training town(s).…”

“…To simplify the driving problem, learning affordances, i.e. learning intermediate representation of the input, is becoming more and more used in both supervised and reinforcement learning for autonomous driving [32,30,27,17]. Indeed, using affordances is extracting useful information, and thus reduces the state space size.…”

“…The agent experiments by itself and thus learns how to act with less human intervention than imitation learning, and also sees a lot more of error cases. With the success of Alpha Go in 2016 [29], reinforcement learning has experienced huge success in machine learning research, but only few approaches are using it for autonomous driving task [22,30,14,16]. Reinforcement learning has indeed several drawbacks such as long training time, poor data efficiency and instability.…”

“…Reinforcement learning has indeed several drawbacks such as long training time, poor data efficiency and instability. However in several domains, such as Go, reinforcement learning has proven to outperform supervised learning, and recent work has established that learning to drive using reinforcement learning could achieve high performance [30,16], even if most of the approaches are still in simulation, for example using the CARLA driving simulator [9]. Yet very often in reinforcement learning framework, the test environment is the same as the training environment, [5], which leaves aside the problem of overfitting and generalization.…”

End-to-End Urban Driving by Imitating a Reinforcement Learning Coach

“…Expert demonstrations are critical for end-to-end AD algorithms. While imitation learning (IL) methods directly mimic the experts' behavior [3,10], reinforcement learning (RL) methods often use expert demonstrations to improve sample efficiency by pre-training part of the model via supervised learning [27,47]. In general, expert demonstrations can be divided into two categories: (i) Off-policy, where the expert directly controls the system, and the state/observation distribution follows the expert.…”