A Policy Search Method For Temporal Logic Specified Reinforcement Learning Tasks

Abstract

Reward engineering is an important aspect of reinforcement learning. Whether or not the users' intentions can be correctly encapsulated in the reward function can significantly impact the learning outcome. Current methods rely on manually crafted reward functions that often requires parameter tuning to obtain the desired behavior. This operation can be expensive when exploration requires systems to interact with the physical world. In this paper, we explore the use of temporal logic (TL) to specify tasks in reinforcement learning. TL formula can be translated to a real-valued function that measures its level of satisfaction against a trajectory. We take advantage of this function and propose temporal logic policy search (TLPS), a model-free learning technique that finds a policy that satisfies the TL specification. A set of simulated experiments are conducted to evaluate the proposed approach.

Arithmetic-Geometric Mean Robustness for Control from Signal Temporal Logic Specifications

“…where λ penalizes the trade-off between maximizing robustness to get the highest STL satisfaction and minimizing the associated cost. Assuming the cost function J is also smooth, a similar gradient ascent optimization can be used to solve the constrained nonlinear optimization problem (16).…”

Prescribed Performance Control Guided Policy Improvement for Satisfying Signal Temporal Logic Tasks

“…A similar problem has been formulated and examined for (a broader range of) completely unknown system dynamics in the context of truncated linear temporal logic (TLTL), a language comparable to STL, by [13]. Therein, the goal was to find a policy that maximizes the expected robustness measure corresponding to a general TLTL task specification.…”

“…Namely, this will be done by using the PPC control law introduced in section II-C to guide PI 2 for an increased rate of convergence and robustness to process noise. We also extend the PI 2 framework to allow optimizing system trajectories subject to STL tasks for general C(τ ) costs; task satisfaction is thus treated as a constraint instead of as the target of optimization, in contrast to [13]. So far, the approach applies to the range of system dynamics (1) and STL formulas (7) to which the discussed PPC control law is applicable.…”

“…A significant difference between our outlined algorithm and [14] is that here a single cost and weight is assigned to each trajectory in its entirety, in contrast to evaluating a cost-to-go at each time step. This difference was also present in the context of TLPS [13]. Extending the definition of the robustness metric associated with satisfying a given STL formula to parts of the trajectory would allow an implementation of PI 2 in its entirety and it would be interesting to see if the result yields any improvements.…”

“…For the purpose of task satisfaction, an STL description of the task becomes suitable because STL is equipped with various robustness measures that quantify the degree of its satisfaction for a given system trajectory in its entirety [12]. Therefore, these measures inherently constitute a descriptive reward to be maximized for task satisfaction and have been shown to be effective for trajectory-based RL methods such as temporal logic policy search (TLPS) [13]. TLPS is based on the policy improvement with path integrals (PI 2 ) algorithm [14], which is applicable to continuous state and action spaces and is a form of sampling-based methods also studied for solving linear temporal logic tasks [15].…”