Maximum Likelihood Constraint Inference for Inverse Reinforcement Learning

Abstract

While most approaches to the problem of Inverse Reinforcement Learning (IRL) focus on estimating a reward function that best explains an expert agent's policy or demonstrated behavior on a control task, it is often the case that such behavior is more succinctly described by a simple reward combined with a set of hard constraints. In this setting, the agent is attempting to maximize cumulative rewards subject to these given constraints on their behavior. We reformulate the problem of IRL on Markov Decision Processes (MDPs) such that, given a nominal model of the environment and a nominal reward function, we seek to estimate state, action, and feature constraints in the environment that motivate an agent's behavior. Our approach is based on the Maximum Entropy IRL framework, which allows us to reason about the likelihood of an expert agent's demonstrations given our knowledge of an MDP. Using our method, we can infer which constraints can be added to the MDP to most increase the likelihood of observing these demonstrations. We present an algorithm which iteratively infers the Maximum Likelihood Constraint to best explain observed behavior, and we evaluate its efficacy using both simulated behavior and recorded data of humans navigating around an obstacle.

Inverse reinforcement learning through logic constraint inference

“…The number of iterations η is a hyperparameter. First, the optimal policy is learned in the nominal (unconstrained) MDP M. From this nominal policy and the set of expert trajectories, one state-action pair is selected by the principle of maximum likelihood [7] and added to the set of constraints C. This is the state which has the highest likelihood under the nominal policy but does not occur in any expert trajectory. Thus, a state which is not visited by an expert but will likely be visited by an agent which is not aware of the implicit rules in the environment.…”

“…Since the set of observed trajectories is finite, the absence of a state-action pair in the observations does not necessarily imply that this pair is invalid. To infer the state-action pairs which are most likely invalid, we build on the principle of maximum likelihood constraint inference [7]. More formally, Scobee et al propose an iterative process where each iteration a constraint c * ∈ C is added to the set of constraints C. This is the constraint which, when augmented on the MDP, maximizes the likelihood of the observed trajectories.…”

“…To prevent transitioning to an empty state, state-action pairs (s, a) for which P trans (s | s, a) > 0 and s ∈ S empty are also added to the set of constraints. Earlier work [7,8] assumes the reward function, which is required to calculate the policy, is known or calculated using an IRL method.…”

“…However, solving this class of constrained MDP's is not trivial and research is still ongoing [20,21]. Scobee et al [7] proposed an alternative definition of constrained MDP's where constraints are defined as invalid state-action pairs, this relaxes the necessity of sophisticated methods to solve the MDP. Hence, the definition of Scobee et al was evaluated to be best suited for this work.…”

“…A promising direction is to learn social norms from human observations. To this end, recent studies have shown principles of inverse reinforcement learning [6] can be adopted to infer both reward function and constraints from observed trajectories [7,8].…”