A Dynamic Regret Analysis and Adaptive Regularization Algorithm for On-Policy Robot Imitation Learning

Abstract

On-policy imitation learning algorithms such as DAgger evolve a robot control policy by executing it, measuring performance (loss), obtaining corrective feedback from a supervisor, and generating the next policy. As the loss between iterations can vary unpredictably, a fundamental question is under what conditions this process will eventually achieve a converged policy. If one assumes the underlying trajectory distribution is static (stationary), it is possible to prove convergence for DAgger. However, in more realistic models for robotics, the underlying trajectory distribution is dynamic because it is a function of the policy. Recent results show it is possible to prove convergence of DAgger when a regularity condition on the rate of change of the trajectory distributions is satisfied. In this article, we reframe this result using dynamic regret theory from the field of online optimization and show that dynamic regret can be applied to any on-policy algorithm to analyze its convergence and optimality. These results inspire a new algorithm, Adaptive On-Policy Regularization (Aor), that ensures the conditions for convergence. We present simulation results with cart-pole balancing and locomotion benchmarks that suggest Aor can significantly decrease dynamic regret and chattering as the robot learns. To our knowledge, this the first application of dynamic regret theory to imitation learning.

Online Learning with Continuous Variations: Dynamic Regret and Reductions

“…We choose not to pursue algorithms with fast static regret rates in COL, as there have been studies on how algorithms can systematically leverage continuity in COL to accelerate learning (Cheng et al, 2019 though they are disguised as online IL research. On the contrary, the knowledge about dynamic regret is less known, except for Lee et al, 2018) (also disguised as online IL) which study the convergence of FTL and mirror descent, respectively.…”

“…An early analysis of IL was framed using the adversarial, static regret setup (Ross et al, 2011). Recently, results were refined through the use of continuity in the bifunction and dynamic regret Lee et al, 2018;Cheng et al, 2019). This problem again highlights the importance of treating stochasticity as the feedback.…”

Continuous Online Learning and New Insights to Online Imitation Learning

“…Using these insights of COL, we revisit online imitation learning (IL) [4] and show it can be framed as a COL problem. We demonstrate that, by using standard analyses of COL, we are able to recover and improve existing understanding of online IL algorithms [4,5,6]. In particular, we characterize existence and uniqueness of solutions, and present convergence and dynamic regret bounds for a common class of IL algorithms in deterministic and stochastic settings.…”

“…The use of online learning to analyze online IL is well established [4]. As studied in [5,6], these online losses can be formulated through a bifunction formulation, l n (π) = f πn (π) = E s∼d πn [c(s, π; π ⋆ )], and the policy class Π can be viewed as the decision set X . Naturally, this online learning formulation results in many online IL algorithms resembling standard online learning algorithms, such as follow-the-leader (FTL), which uses full information feedback l n (•) = E s∼d πn [c(s, •; π ⋆ )] at each round, [4] and mirror descent [23], which uses the first-order feedback…”

“…The original work by Ross et al [4] analyzed FTL in the static regret case by immediate reductions to known static regret bounds of FTL. However, a crucial objective is understanding when these algorithms converge to useful solutions in terms of policy performance, which more recent work has attempted to address [5,6,24]. According to these refined analyses, dynamic regret is a more appropriate solution concept to online IL when π ⋆ / ∈ Π, which is the common case in practice.…”