Trajectory Optimization for Unknown Constrained Systems using Reinforcement Learning

Abstract

In this paper, we propose a reinforcement learning-based algorithm for trajectory optimization for constrained dynamical systems. This problem is motivated by the fact that for most robotic systems, the dynamics may not always be known. Generating smooth, dynamically feasible trajectories could be difficult for such systems. Using samplingbased algorithms for motion planning may result in trajectories that are prone to undesirable control jumps. However, they can usually provide a good reference trajectory which a model-free reinforcement learning algorithm can then exploit by limiting the search domain and quickly finding a dynamically smooth trajectory. We use this idea to train a reinforcement learning agent to learn a dynamically smooth trajectory in a curriculum learning setting. Furthermore, for generalization, we parameterize the policies with goal locations, so that the agent can be trained for multiple goals simultaneously. We show result in both simulated environments as well as real experiments, for a 6-DoF manipulator arm operated in position-controlled mode to validate the proposed idea. We compare the proposed ideas against a PID controller which is used to track a designed trajectory in configuration space. Our experiments show that our RL agent trained with a reference path outperformed a model-free PID controller of the type commonly used on many robotic platforms for trajectory tracking.

Reinforcement learning for robot research: A comprehensive review and open issues

“…Ota et al proposed that a 6-DoF manipulator arm trained with a good reference trajectory to quickly track a designed trajectory in configuration space. 122 In a recent study, an improved Q-learning algorithm is exploited to form a reward and penalty mechanism, 123 which effectively tackles the problem of robotic time-optimal route tracking with prior knowledge. The prior knowledge of leg trajectories was embedded into the action space during safe exploration with only less data collection to achieve walking on a quadruped robot.…”

OPIRL: Sample Efficient Off-Policy Inverse Reinforcement Learning via Distribution Matching

“…Mode-covering can be achieved via forward KL-Divergence, such as behavior cloning. Hence, we incorporate Behavior Cloning Loss [21,34] into our policy updates to encourage active exploration at the beginning stages of training. We also add Q-Filter to avoid sub-optimal actions being chosen.…”

Deep Reactive Planning in Dynamic Environments

“…The combination of reference paths and RL has been widely researched [9,10,11,12]. In [9,11], Probabilistic Roadmaps (PRM) [2] are used to find reference paths, and RL is used for point-to-point navigation as a local planner for PRM.…”

“…In our proposed work, this is achieved as the path encodes the goal and automatically tries to avoid obstacles in the environment, too. The closest work to ours is [12,18]. [12] learns an RL agent that optimizes trajectory for a 6-DoF manipulator arm.…”

“…The closest work to ours is [12,18]. [12] learns an RL agent that optimizes trajectory for a 6-DoF manipulator arm. Our approach, however, can deal with changes in the environment by conditioning an RL agent with a reference path which is also generated based on an observation of the current environment.…”