Hierarchical reinforcement learning for self‐driving decision‐making without reliance on labelled driving data

Abstract

Decision making for self-driving cars is usually tackled by manually encoding rules from drivers' behaviors or imitating drivers' manipulation using supervised learning techniques. Both of them rely on mass driving data to cover all possible driving scenarios. This paper presents a hierarchical reinforcement learning method for decision making of self-driving cars, which does not depend on a large amount of labeled driving data. This method comprehensively considers both high-level maneuver selection and low-level motion control in both lateral and longitudinal directions. We firstly decompose the driving tasks into three maneuvers, including driving in lane, right lane change and left lane change, and learn the sub-policy for each maneuver. Then, a master policy is learned to choose the maneuver policy to be executed in the current state. All policies including master policy and maneuver policies are represented by fully-connected neural networks and trained by using asynchronous parallel reinforcement learners (APRL), which builds a mapping from the sensory outputs to driving decisions. Different state spaces and reward functions are designed for each maneuver. We apply this method to a highway driving scenario, which demonstrates that it can realize smooth and safe decision making for self-driving cars.

Evaluation Model of Autonomous Vehicles’ Speed Suitability Based on Overtaking Frequency

“…Global domain experts have also researched the factors that need to be considered for driverless automobiles to make decisions. Duan proposed a hierarchical reinforcement learning method for self-driving vehicle decisions, which was applied to a highway driving scene [ 13 ]. The Carnegie Mellon Navlab vehicles combine a rationally distributed system (PolySAPIENT) with a novel evolutionary optimization strategy (PBIL) and use route-level planning to achieve context-sensitive local decision-making and complex motion planning, using reasonable behavior theory to decide driving behavior [ 14 , 15 , 16 , 17 ].…”

A centralised training algorithm with D3QN for scalable regular unmanned ground vehicle formation maintenance

“…Compared to classic autonomous drive systems with environment perception, path planning, and dynamics control modules [11], the DRL method combines deep neural networks with a reinforcement learning frame. This combination can help UGV achieve a faster response to more complex traffic scenarios by learning strategies from high-dimensional perceptual input through the end-to-end model [12]. Therefore, the application of DRL in the UGV field is attracting an increasing research focus.…”

Safe-update of bi-layered controller and its application to power systems

“…Some works address decentralized structure in the controller and/or apply data-based controller design. Although the works [4,5] show the effectiveness of decentralized control and data-based design, it is not always realistic to completely replace the existing reliable controller with a new one. As is pointed out in [6,7], the safety of the control systems may be impaired during the update of the controller.…”