Policy space identification in configurable environments

Abstract

We study the problem of identifying the policy space available to an agent in a learning process, having access to a set of demonstrations generated by the agent playing the optimal policy in the considered space. We introduce an approach based on frequentist statistical testing to identify the set of policy parameters that the agent can control, within a larger parametric policy space. After presenting two identification rules (combinatorial and simplified), applicable under different assumptions on the policy space, we provide a probabilistic analysis of the simplified one in the case of linear policies belonging to the exponential family. To improve the performance of our identification rules, we make use of the recently introduced framework of the Configurable Markov Decision Processes, exploiting the opportunity of configuring the environment to induce the agent to reveal which parameters it can control. Finally, we provide an empirical evaluation, on both discrete and continuous domains, to prove the effectiveness of our identification rules.

Sample Dropout: A Simple yet Effective Variance Reduction Technique in Deep Policy Optimization

“…We also demonstrate in our experiments that our sample dropout technique can boost the sample efficiency of GAE-based policy optimization algorithms. [28] and [29] propose an actor-only policy optimization algorithm that alternates online and offline optimization via important sampling. To capture the uncertainty induced by importance sampling, they propose a surrogate objective function derived from a statistical bound on the estimated performance, which helps bound the variance of surrogate objective in terms of the Renyi divergence.…”

Optimizing Empty Container Repositioning and Fleet Deployment via Configurable Semi-POMDPs

“…Conf-MDPs have been introduced in [7] for finite spaces, and extended in [9] for more complex continuous environments. In these seminal works, the agent is fully responsible for the configuration activity of the environment, which, in turn, results in an auxiliary task to optimize performance.…”

“…Indeed, in Conf-MDPs, the agent is not interested in learning and gathering experience samples in sub-optimal configuration; its interest is solely toward the optimal policy in the optimal environmental configuration. The configuration activity within the environment, as shown in more recent works [8], [11], can also be carried out by an external entity (i.e., configurator) whose goals can even be adversary w.r.t. the ones of the agent [11].…”

“…More specifically, from an agents' perspective, the fleet of vessels can be seen as features of the environment that can be optimized to reach higher performances. In this sense, for single-agent problems, Configurable Markov Decision Processes (Conf-MDPs) [7]- [9] have recently been introduced to extend the Markov Decision Process (MDP) [10] framework to account for environmental configurations. In Conf-MDPs, an agent and a configurator are responsible for finding the optimal policy-configuration pair.…”

“…This is clearly related to our application scenario: our agents are in charge of deciding which container repositioning policy to play, whereas the configurator is entitled to select the fleet of vessels. While the early works [7]- [9] focused on the case in which agent and configurator share the same objective, in [11], the setting has been extended to the case in which the configurator and the agent have different (and, possibly, adversarial) goals. Although these approaches have strong theoretical guarantees, how to successfully scale them to more complex domains remains an open question.…”