Policy Optimization with Linear Temporal Logic Constraints

Abstract

We study the problem of policy optimization (PO) with linear temporal logic (LTL) constraints. The language of LTL allows flexible description of tasks that may be unnatural to encode as a scalar cost function. We consider LTL-constrained PO as a systematic framework, decoupling task specification from policy selection, and an alternative to the standard of cost shaping. With access to a generative model, we develop a model-based approach that enjoys a sample complexity analysis for guaranteeing both task satisfaction and cost optimality (through a reduction to a reachability problem). Empirically, our algorithm can achieve strong performance even in low sample regimes.Preprint. Under review.

Model-free Neural Lyapunov Control for Safe Robot Navigation

“…However, this prior work has not considered how to synergistically take advantage of policies with differentiable [13], [14] and programmable [30], [31], [32] formal constraints whose definitions can directly encode the high-level tasks. More recently, a constrained reinforcement learning framework with formal constraints [11] was proposed and provided a theoretical foundation for constrained reinforcement learning under formal constraints in a discrete state and action space. However, this work does not consider directly backpropagating through formal constraints and requires learning high-level abstractions, making it perform modestly in the high-dimensional continuous control tasks we consider.…”

“…Logic-Constrained Reinforcement Learning (LCRL) [1], [2], [3], [4], [5], [6], [7], [8], [9], [10], [11] is a rapidly growing field that enables end-to-end reinforcement learning to perform various high-level tasks. For example, a logical task constraint can require a drone to hover near a doorway around until the door is opened, execute a collection of complex instructions sequentially, or make decisions that are tailored to different conditions.…”