Safe Reinforcement Learning with Scene Decomposition for Navigating Complex Urban Environments

Abstract

Navigating urban environments represents a complex task for automated vehicles. They must reach their goal safely and efficiently while considering a multitude of traffic participants. We propose a modular decision making algorithm to autonomously navigate intersections, addressing challenges of existing rule-based and reinforcement learning (RL) approaches. We first present a safe RL algorithm relying on a model-checker to ensure safety guarantees. To make the decision strategy robust to perception errors and occlusions, we introduce a belief update technique using a learning based approach. Finally, we use a scene decomposition approach to scale our algorithm to environments with multiple traffic participants. We empirically demonstrate that our algorithm outperforms rule-based methods and reinforcement learning techniques on a complex intersection scenario.

Safe Reinforcement Learning for Autonomous Lane Changing Using Set-Based Prediction

“…One method to alter the action selection process is to prioritize actions that are estimated to be safer [3]. However, this approach does not prove the nonexistence of unsafe behaviors.…”

“…Various approaches have been proposed to increase the safety of RL methods by modifying the optimality criterion [1], [2] or by verifying the exploration processes with external guidance [3]- [10]. By modifying the optimality objective, agents behave more cautious than those trained without a risk measure included in the objective; however, the absence of unsafe behaviors cannot be proven.…”

On the Impact of the Rules on Autonomous Drive Learning

“…A different strategy is followed by [35], where the authors defined a custom set of traffic rules based on the environment, the driver, and the road graph. With these rules, a RL driver learns to safely make lane-changing decisions, where the driver's decision making is combined with the formal safety verification of the rules, to ensure that only safe actions are taken by the driver A similar approach is considered in [7], where the authors replaced the formal safety verification with a learnable safety belief module, as part of the driver's policy.…”

“…The use of simulations and synthetic data [5] for training have allowed to assess neural networks capabilities in many different realistic environments and different degrees of complexity. Many driving simulators have been designed, from the low-level ones that allow the drivers to control the hand brake of their car [6], to higher-level ones, in which the drivers can control their car acceleration and lane-change [7]. Some simulators model the traffic in an urban road network [8], some others model car's intersection access [9][10][11][12], or roundabout insertion [13].…”

Cooperation-Aware Reinforcement Learning for Merging in Dense Traffic

“…RL provides a flexibility in the choice of the interaction model. RL has been applied to a variety of driving scenarios such as lane changing [10], or intersection navigation [11], [12].…”