Reinforcement Learning of Control Policy for Linear Temporal Logic Specifications Using Limit-Deterministic Generalized Büchi Automata

Abstract

This letter proposes a novel reinforcement learning method for the synthesis of a control policy satisfying a control specification described by a linear temporal logic formula. We assume that the controlled system is modeled by a Markov decision process (MDP). We convert the specification to a limit-deterministic generalized Büchi automaton (LDGBA) with several accepting sets that accepts all infinite sequences satisfying the formula. The LDGBA is augmented so that it explicitly records the previous visits to accepting sets. We take a product of the augmented LDGBA and the MDP, based on which we define a reward function. The agent gets rewards whenever state transitions are in an accepting set that has not been visited for a certain number of steps. Consequently, sparsity of rewards is relaxed and optimal circulations among the accepting sets are learned. We show that the proposed method can learn an optimal policy when the discount factor is sufficiently close to one.

Modular Deep Reinforcement Learning for Continuous Motion Planning with Temporal Logic

“…To facilitate learning of optimal policies, the designed reward is enhanced with potential functions that effectively guide the agent toward task satisfaction without adding extra hyper-parameters to the algorithm. Unlike [18], rigorous analysis shows that the maximum probability of task satisfaction can be guaranteed. Compared to approaches based on limit deterministic Büchi automata (LDBA), e.g., [16,17], LDGBA has several accepting sets while LDBA only has one accepting set which can result in sparse rewards during training.…”

“…We assign each DDPG an individual replay buffer B qi and a random process noise N qi . The corresponding weights of modular networks, i.e., Q qi x, u P θ Q q i and π qi (x |θ uq i ), are also updated at each iteration (line [15][16][17][18][19][20]. All neural networks are trained using their own replay buffer, which is a finite-sized cache that stores transitions sampled from exploring the environment.…”

“…However, scalability is a pressing issue for applying model-based approaches due to the need to store the learned model. On the other hand, by relaxing the need to construct an MDP model, model-free RL is recently adopted where appropriate reward shaping schemes are proposed [12][13][14][15][16][17][18][19][20].…”

“…q 12 = q 3 . However, for x 1 = ( s , q 1 , T ) and x 2 = ( s , q 2 , T ), we have Φ (x 1 ) = 0 and Φ (x 2 ) = 0 if the corresponding automaton component has not been visited yet (i.e., q 1 , q 2 still in T Φ ) by ( 17) and (18). For instance, given a run x = ( s , q 0 , T ) u P 0 ( s , q 1 , T ) u P 1 ( s , q 2 , T ) u P 2 ( s , q 3 , T ) , the associated shaped reward for each transition is R ( s , q 0 , T ) , u P 0 , ( s , q…”

Seminator 2 Can Complement Generalized Büchi Automata via Improved Semi-determinization

“…This latter perspective has recently initiated another wave of research on semi-deterministic automata. Since 2015, many new results have been published: several direct translations of LTL to semideterministic automata [11,15,16,26], specialized complementation constructions for semi-deterministic automata [4,6], algorithms for quantitative model checking of MDPs based on semi-deterministic automata [13,25], a transformation of semideterministic automata to deterministic parity automata [10], and reinforcement learning of control policy using semi-deterministic automata [21].…”