Joint Inference of Reward Machines and Policies for Reinforcement Learning

Abstract

Incorporating high-level knowledge is an effective way to expedite reinforcement learning (RL), especially for complex tasks with sparse rewards. We investigate an RL problem where the high-level knowledge is in the form of reward machines, a type of Mealy machines that encode non-Markovian reward functions. We focus on a setting in which this knowledge is a priori not available to the learning agent. We develop an iterative algorithm that performs joint inference of reward machines and policies for RL (more specifically, q-learning). In each iteration, the algorithm maintains a hypothesis reward machine and a sample of RL episodes. It uses a separate q-function defined for each state of the current hypothesis reward machine to determine the policy and performs RL to update the q-functions. While performing RL, the algorithm updates the sample by adding RL episodes along which the obtained rewards are inconsistent with the rewards based on the current hypothesis reward machine. In the next iteration, the algorithm infers a new hypothesis reward machine from the updated sample. Based on an equivalence relation between states of reward machines, we transfer the q-functions between the hypothesis reward machines in consecutive iterations. We prove that the proposed algorithm converges almost surely to an optimal policy in the limit. The experiments show that learning high-level knowledge in the form of reward machines leads to fast convergence to optimal policies in RL, while the baseline RL methods fail to converge to optimal policies after a substantial number of training steps.

Active Grammatical Inference for Non-Markovian Planning

“…Task 3 is the Minecraft-like gridworld introduced in (Andreas, Klein, and Levine 2017), where the objective is to build a spear by gathering wood, string, and stone in any order, then reaching a workbench (See Figure 3 in (Andreas, Klein, and Levine 2017)). We evaluate the time and sample efficiency of our non-Markovian planning method compared to the JIRP and AFRAI methods presented in (Xu et al 2020) and (Xu et al 2021), respectively. A sample in our method includes the queries to the model during learning and the environment samples observed during execution.…”

“…Active inference allows for the system under learning to be queried, whereas its passive counterpart leverages an existing static repository of samples. The work in (Xu et al 2020) uses passive grammatical inference by storing traces of behavior that arise in the standard Q-learning methodology. This repository can then be used to synthesize a reward machine using techniques like satisfiability solving.…”

“…In recent years, this has been explored in the context of RL, where access to the model is not available and the agent must learn through black-box exploration of the environment. In this setting, quintessential algorithms from the field of grammatical inference have been applied with some success (Gaon and Brafman 2020;Xu et al 2020Xu et al , 2021. However, non-Markovian planning with known transition dynamics remains largely unexplored (with the exception of the pre-print in (Rens and Raskin 2020)).…”

Verifiable and Compositional Reinforcement Learning Systems

“…Sarathy et al (2021) incorporates RL with symbolic planning models to learn new operators -similar to our subtasksto aid in the completion of planning objectives. Meanwhile, Toro Icarte et al (2018; Xu et al (2020);Toro Icarte et al (2022) use reward machines, finite-state machines encoding temporally extended tasks in terms of atomic propositions, to break tasks into stages for which separate policies can be learned. Neary et al (2021b) extends the use of reward machines to the multi-agent RL setting, decomposing team tasks into subtasks for individual learners.…”

Inferring Probabilistic Reward Machines from Non-Markovian Reward Signals for Reinforcement Learning

“…While much progress has been made on learning and leveraging reward machines for decision processes with non-Markovian rewards (Xu et al 2020(Xu et al , 2021Abadi and Brafman 2020;Gaon and Brafman 2020;Neider et al 2021;Rens et al 2021), the more general setting where rewards exhibit both non-Markovian and stochastic dynamics has not been addressed. In this paper, we make progress on this front by introducing probabilistic reward machines (PRMs).…”