Maximum margin planning

Abstract

Imitation learning of sequential, goaldirected behavior by standard supervised techniques is often difficult. We frame learning such behaviors as a maximum margin structured prediction problem over a space of policies. In this approach, we learn mappings from features to cost so an optimal policy in an MDP with these cost mimics the expert's behavior. Further, we demonstrate a simple, provably efficient approach to structured maximum margin learning, based on the subgradient method, that leverages existing fast algorithms for inference. Although the technique is general, it is particularly relevant in problems where A* and dynamic programming approaches make learning policies tractable in problems beyond the limitations of a QP formulation. We demonstrate our approach applied to route planning for outdoor mobile robots, where the behavior a designer wishes a planner to execute is often clear, while specifying cost functions that engender this behavior is a much more difficult task.

Integrating human observer inferences into robot motion planning

“…Here, the robot can learn a C that explains the demonstrations [4], using tools like Inverse Optimal Control (IOC) [1,44,60]. However, extending these tools to higher dimensions is an open problem [44], and recent work focuses on learning costs that make the demonstrations locally optimal [32,38], or on restricting the space of trajectories to one in which optimization is tractable [30].…”

Learning the Combinatorial Structure of Demonstrated Behaviors with Inverse Feedback Control

“…In opposition, inverse optimal control [14] and inverse reinforcement learning [15] are approaches based on the idea that, in some situations, it can lead to better generalization to model aspects of the task that the demonstrator is trying to solve instead of modeling the particular solution in the demonstrated context. The capacity of inverse optimal control to achieve better generalization has been demonstrated in thee x p e r i m e n tp e r f o r m e db yA b b e e le ta l .…”

Training parsers by inverse reinforcement learning

“…In fact, a parse in this framework is obtained as the trajectory when an optimal policy is followed in an appropriately defined MDP. This idea is not completely new: The reverse connection was exploited by Ratliff et al (2006) who derived an algorithm for inferring rewards using the large margin approach of Taskar et al (2005). Maes et al (2007) have used reinforcement learning for solving the structured prediction problem of sequence labeling.…”

“…The fourth algorithm considered is the Max-Margin Planning method of Ratliff et al (2006) which has already been mentioned previously. This algorithm uses the same criterion as Taskar et al (2004), but instead of using the so-called structured SMO method used by Taskar et al (2004), following the suggestion of Ratliff et al (2006) we implement the optimizer using a subgradient method.…”

“…The fourth algorithm considered is the Max-Margin Planning method of Ratliff et al (2006) which has already been mentioned previously. This algorithm uses the same criterion as Taskar et al (2004), but instead of using the so-called structured SMO method used by Taskar et al (2004), following the suggestion of Ratliff et al (2006) we implement the optimizer using a subgradient method. The last algorithm is the recently proposed IRL method of Ziebart et al (2008) which turns out to be a close relative to the Maximum Entropy discriminative reranking method proposed by Charniak and Johnson (2005).…”