Reinforcement Learning for Temporal Logic Control Synthesis with Probabilistic Satisfaction Guarantees

Abstract

Reinforcement Learning (RL) has emerged as an efficient method of choice for solving complex sequential decision making problems in automatic control, computer science, economics, and biology. In this paper we present a model-free RL algorithm to synthesize control policies that maximize the probability of satisfying high-level control objectives given as Linear Temporal Logic (LTL) formulas. Uncertainty is considered in the workspace properties, the structure of the workspace, and the agent actions, giving rise to a Probabilistically-Labeled Markov Decision Process (PL-MDP) with unknown graph structure and stochastic behaviour, which is even more general case than a fully unknown MDP. We first translate the LTL specification into a Limit Deterministic Büchi Automaton (LDBA), which is then used in an on-the-fly product with the PL-MDP. Thereafter, we define a synchronous reward function based on the acceptance condition of the LDBA. Finally, we show that the RL algorithm delivers a policy that maximizes the satisfaction probability asymptotically. We provide experimental results that showcase the efficiency of the proposed method.

Reinforcement Learning of Control Policy for Linear Temporal Logic Specifications Using Limit-Deterministic Generalized Büchi Automata

“…Any LTL formula ϕ can be converted into various ωautomata, namely finite state machines that recognize all infinite words satisfying ϕ. We review a generalized Büchi automaton at the beginning, and then introduce a limitdeterministic generalized Büchi automaton [10].…”

“…Through the above scenario, we compare our approach with 1) a case where we first convert the tLDGBA into a tLDBA, for which the augmentation makes no change, and thus a reward function in Definition 10 is based on a single accepting set; and 2) the method using a reward function based on the accepting frontier function [9], [10]. For the three methods, we use Qlearning 1 with an epsilon-greedy policy.…”

“…5. The optimal policy obtained from our proposed method (left) and the method in [9], [10] (right). Fig.…”

“…2) We use the accepting frontier function [9], [10] for the tLDGBA Acc : δ × 2 δ → 2 δ . Initializing a set of transitions F with the set of the all accepting transitions in B ϕ , the function receives the transition (x, σ, x ) that occurs and the set F. If (x, σ, x ) is in F, then Acc removes the accepting sets containing (x, σ, x ) from F. For the product MDP of the MDP M and the tLDGBA B ϕ , the reward function is based on the removed sets of B ϕ .…”

“…The conversion, however, fixes the order of visits to accepting sets of the GBA [3] and causes the sparsity of the reward, which is a critical issue in RL-based controller synthesis. Another approach to RL-based synthesis for generalized Büchi conditions is the accepting frontier function introduced in [9], [10], based on which the reward function is defined. However, the function is memoryless, that is, it does not provide information of accepting sets that have been visited, which is important to improve learning performance.…”