What if the World Were Different? Gradient-Based Exploration for New Optimal Policies

Abstract

Planning under uncertainty assumes a model of the world that specifies the probabilistic effects of the actions of an agent in terms of changes of the state. Given such model, planning proceeds to determine a policy that defines for each state the choice of action that the agent should follow in order to maximize a reward function. In this work, we realize that the world can be changed in more ways than those possible by the execution of the agent’s repertoire of actions. These additional configurations of the world may allow new policies that let the agent accumulate even more reward than that possible by following the optimal policy of the original world. We introduce and formalize the problem of planning while considering these additional possible worlds. We then present an approach that models feasible changes to the world as modifications to the probability transition function, and show that the problem of computing the configuration of the world that allows the most rewarding optimal policy can be formulated as a constrained optimization problem. Finally, we contribute a gradient-based algorithm for solving this optimization problem. Experimental evaluation shows the effectiveness of our approach in multiple problems of practical interest.

A Theoretical and Algorithmic Analysis of Configurable MDPs

“…where, for compactness, given a policy π and world P we define µ π P = µ 0 (I − γP π ) −1 . This matches the gradient step of the method proposed previously by Silva, Melo, and Veloso (2018). The gradient step of this method was introduced as an approximation.…”

“…Recently, we have seen a shift in this paradigm, with new approaches that allow the agent to explicitly reason at a "meta-level" about other possible configurations of the world-those configurations that are achievable indirectly, through changes of environmental features controllable only before planning time. Experimental evaluation of this new paradigm showed promising results on different planning scenarios modeled as Markov decision processes (Metelli, Mutti, and Restelli 2018;Silva, Melo, and Veloso 2018).…”

“…Their approach optimizes over the convex hull of that set of world configurations, searching for the configuration that maximizes the expected rewards. Silva, Melo, and Veloso (2018), on the other hand, model possible changes to the world through a generic parameterization of the transition probabilities, and assume a cost function that penalizes changes to the original world configuration. Their approach uses local information (gradient) to optimize over the given space of parameters, searching for the world configuration that maximizes the trade-off between the expected rewards and the costs associated with the changes to the world.…”

Configurable Environments in Reinforcement Learning: An Overview

“…Instead, the possibility to strategically act on the environmental dynamics is studied in a limited number of works only. Some approaches belonging to the planning area [12,38], some are constrained to specific forms of environment configurability [8,9,34], and others based on the curriculum learning framework [4,7]. The goal of the dissertation [18] is to provide a uniform treatment of environment configurability in its diverse aspects.…”

Policy Space Identification in Configurable Environments

“…The knowledge of the agent's policy space could be of crucial importance when the learning process involves the presence of an external supervisor. Recently, the notion of Configurable Markov Decision Process (Conf-MDP, Metelli, Mutti, and Restelli 2018) has been introduced to account for the real-world scenarios in which it is possible to exercise a, maybe partial, control over the environment, by means of a set of environmental parameters (e.g., Silva, Melo, and Veloso 2018;Silva et al 2019). This activity, called environment configuration, can be carried out by the agent itself or by an external supervisor.…”