Preference-Based Policy Learning

Abstract

Abstract. Many machine learning approaches in robotics, based on reinforcement learning, inverse optimal control or direct policy learning, critically rely on robot simulators. This paper investigates a simulatorfree direct policy learning, called Preference-based Policy Learning (PPL). PPL iterates a four-step process: the robot demonstrates a candidate policy; the expert ranks this policy comparatively to other ones according to her preferences; these preferences are used to learn a policy return estimate; the robot uses the policy return estimate to build new candidate policies, and the process is iterated until the desired behavior is obtained. PPL requires a good representation of the policy search space be available, enabling one to learn accurate policy return estimates and limiting the human ranking effort needed to yield a good policy. Furthermore, this representation cannot use informed features (e.g., how far the robot is from any target) due to the simulator-free setting. As a second contribution, this paper proposes a representation based on the agnostic exploitation of the robotic log. The convergence of PPL is analytically studied and its experimental validation on two problems, involving a single robot in a maze and two interacting robots, is presented.

Preference-based reinforcement learning: evolutionary direct policy search using a preference-based racing algorithm

“…The idea of preference-based reinforcement learning was introduced simultaneously and independently in Akrour et al (2011) and Cheng et al (2011). While preferences on histories are taken as a point of departure in both approaches, policy learning is accomplished in different ways.…”

“…The preference-based policy learning settings considered in Fürnkranz et al (2012), Akrour et al (2011) proceed from a (possibly partial) preference relation over histories h ∈ H (T ) , and the goal is to find a policy which tends to generate preferred histories with high probability. In this regard, it is notable that, in the EDPS framework, the precise values of the function to be optimized (in this case the expected total rewards) are actually not used by the evolutionary optimizer.…”

“…As a direct policy search method, it shares commonalities with Akrour et al (2011), but also differs in several respects. In particular, the latter approach (as well as follow-up work of the same authors, such as Akrour et al (2012)) is specifically tailored for applications in which a user interacts with the learner in an iterative process.…”

“…In Akrour et al (2011) and Cheng et al (2011), the authors tackle the problem of learning policies solely on the basis of qualitative preference information, namely pairwise comparisons between histories; such comparisons suggest that one system behavior is preferred to another one, but without committing to precise numerical rewards. Building on novel methods for preference learning, this is accomplished by providing the RL agent with qualitative policy models, such as ranking functions.…”

“…More specifically, Cheng et al (2011) use a method called label ranking to train a model that ranks actions given a state; their approach generalizes classification-based approximate policy iteration (Lagoudakis and Parr 2003). Instead of ranking actions given states, Akrour et al (2011) learn a preference model on histories, which can then be used for policy optimization.…”