A Framework for Transforming Specifications in Reinforcement Learning

Abstract

Reactive synthesis algorithms allow automatic construction of policies to control an environment modeled as a Markov Decision Process (MDP) that are optimal with respect to high-level temporal logic specifications assuming the MDP model is known a priori. Reinforcement learning algorithms, in contrast, are designed to learn an optimal policy when the transition probabilities of the MDP are unknown, but require the user to associate local rewards with transitions. The appeal of highlevel temporal logic specifications has motivated research to develop RL algorithms for synthesis of policies from specifications. To understand the techniques, and nuanced variations in their theoretical guarantees, in the growing body of resulting literature, we develop a formal framework for defining transformations among RL tasks with different forms of objectives. We define the notion of sampling-based reduction to relate two MDPs whose transition probabilities can be learnt by sampling, followed by formalization of preservation of optimal policies, convergence, and robustness. We then use our framework to restate known results, establish new results to fill in some gaps, and identify open problems.

Specification-Guided Learning of Nash Equilibria with High Social Welfare

“…There has been recent work on using specifications based on temporal logic for specifying RL tasks in the single agent setting; a comprehensive survey may be found in [2]. There has also been recent work on using temporal logic specifications for multi-agent RL [10,22], but these approaches focus on cooperative scenarios in which there is a common objective that all agents are trying to achieve.…”

Policy Optimization with Linear Temporal Logic Constraints

“…where ψ sas (n) ≡ 2 P (s, a, s )(1 − P (s, a, s )))ξ(n) + 7 3 ξ(n), ψ(n) ≡ 1 2 ξ(n) + 7 3 ξ(n), and ξ(n) ≡ log( 4n 2 |S| 2 |A| δ )/(n − 1). 3 Our assumptions are consistent with the minimal requirements studied by [39] Remark 4.1.…”

“…These assumptions have complex interactions with the environment, making them impractical if not impossible to calculate. The situation is made more complex by recent theoretical results [61,7] that show that there exist LTL tasks that are not PAC-MDP-learnable.…”

Specification-Guided Learning of Nash Equilibria with High Social Welfare

“…RL from high-level specifications. There has been recent work on using specifications based on temporal logic for specifying RL tasks in the single agent setting; a comprehensive survey may be found in [2]. There has also been recent work on using temporal logic specifications for multi-agent RL [10,24], but these approaches focus on cooperative scenarios in which there is a common objective that all agents are trying to achieve.…”

“…Here, achieve and ensuring correspond to the "eventually" and "always" operators in temporal logic 2. We do not require that the subgoal regions partition the state space or that they be non-overlapping.…”