Advice-Guided Reinforcement Learning in a non-Markovian Environment

Abstract

We study a class of reinforcement learning tasks in which the agent receives its reward for complex, temporally-extended behaviors sparsely. For such tasks, the problem is how to augment the state-space so as to make the reward function Markovian in an efficient way. While some existing solutions assume that the reward function is explicitly provided to the learning algorithm (e.g., in the form of a reward machine), the others learn the reward function from the interactions with the environment, assuming no prior knowledge provided by the user. In this paper, we generalize both approaches and enable the user to give advice to the agent, representing the user’s best knowledge about the reward function, potentially fragmented, partial, or even incorrect. We formalize advice as a set of DFAs and present a reinforcement learning algorithm that takes advantage of such advice, with optimal con- vergence guarantee. The experiments show that using well- chosen advice can reduce the number of training steps needed for convergence to optimal policy, and can decrease the computation time to learn the reward function by up to two orders of magnitude.

Inferring Probabilistic Reward Machines from Non-Markovian Reward Signals for Reinforcement Learning

“…While most literature assumes a reward machine is given (Icarte et al 2018;Hahn et al 2019), the problem of learning the machine from observations has only been considered recently. The work in (Xu et al 2020;Neider et al 2021) explores a solving approach based on satisfiability. In (Icarte et al 2019), the problem of learning a reward machine is viewed through the lens of discrete optimization.…”

“…However, many problem domains require taking into account this history. Examples include learning in settings with sparse rewards (Neider et al 2021), rewards defined as regular expressions and formal logics (Camacho et al 2019), and decision-making under partial observability (Icarte et al 2019). In such settings, the non-Markovian reward signal may not be known, but can be learned from traces of behavior.…”

“…While much progress has been made on learning and leveraging reward machines for decision processes with non-Markovian rewards (Xu et al 2020(Xu et al , 2021Abadi and Brafman 2020;Gaon and Brafman 2020;Neider et al 2021;Rens et al 2021), the more general setting where rewards exhibit both non-Markovian and stochastic dynamics has not been addressed. In this paper, we make progress on this front by introducing probabilistic reward machines (PRMs).…”

Policy Dispersion in Non-Markovian Environment

“…In most real-world problems, the rewards do not depend on the immediate state and the chosen action but rather on the agent's visited states and performed actions. In such environments, the Markovian assumption (MA) does not hold, and the reward function has a temporal nature [31], that the agent receives its rewards for complex, temporallyextended behaviors sparsely. For example, a robot should be rewarded for delivering coffee only if a user previously requested it.…”

“…We use a Transformer-based architecture [36] to capture long-horizon dependencies for histories of state-action pairs, which include all reward-relevant historical information. Moreover, in non-Markovian environments, the agent receives its reward sparsely for complex actions over a long period of time [31], which is not conducive to training the Transformerbased policy representation model. We construct a sample categorical distribution to sample higher cumulative reward trajectories with a higher probability.…”