Trajectory Planning with Deep Reinforcement Learning in High-Level Action Spaces

Abstract

This paper presentsa technique for trajectory planning based on continuously parameterized high-level actions (motion primitives) of variable duration. This technique leverages deep reinforcement learning (Deep RL) to formulate a policy which is suitable for real-time implementation. There is no separation of motion primitive generation and trajectory planning: each individual short-horizon motion is formed during the Deep RL training to achieve the full-horizon objective. Effectiveness of the technique is demonstrated numerically on a well-studied trajectory generation problem and a planning problem on a known obstacle-rich map. This paper also develops a new loss function term for policy-gradient-based Deep RL, which is analogous to an anti-windup mechanism in feedback control. We demonstrate the inclusion of this new term in the underlying optimization increases the average policy return in our numerical example.

Formal Certification Methods for Automated Vehicle Safety Assessment

“…Under such a context, continuous feasibility and control liveliness are key concerns ensuring the accomplishment of higher level tasks, such as overtaking another slow vehicle, merging into a busy fast lane, or avoiding an obstacle at high speeds [117]. When selecting mode from a large library of complicated dynamic modes become computationally demanding, learning based approach such as reinforcement learning can be leveraged to efficiently learn the proper mode arbitration decision according to the environment [118], [119].…”

Noise- and Fuel-Minimal Departure Trajectory Optimization with Reinforcement Learning

“…One possible solution is to apply Deep Reinforcement Learning (RL), which has demonstrated its effectiveness in resolving a variety of optimal control problems across different őelds, such as autonomous driving and game playing. As one of emerging technologies, Deep RL exhibits great promises to enhance the efficiency of trajectory planning [15,16], especially in systems with complicated constraints and dynamics. It also demonstrates the ability to maintain good performance in stochastic environments [17].…”

“…Its thoughtfully engineered reward shaping is one main factor in taking full advantage of its Deep RL algorithmÐSoft Actor-Critic (SAC) [21]. A recent study by Williams et al [15] also incorporates signs of motion primitives. However, instead of creating a non-intuitive motion primitive planning, they proposed a technique called high-level action space, which allows variable time-stepping in controlling the vehicle.…”