Unifying Gradient Estimators for Meta-Reinforcement Learning via Off-Policy Evaluation

Abstract

Model-agnostic meta-reinforcement learning requires estimating the Hessian matrix of value functions. This is challenging from an implementation perspective, as repeatedly differentiating policy gradient estimates may lead to biased Hessian estimates. In this work, we provide a unifying framework for estimating higherorder derivatives of value functions, based on off-policy evaluation. Our framework interprets a number of prior approaches as special cases and elucidates the bias and variance trade-off of Hessian estimates. This framework also opens the door to a new family of estimates, which can be easily implemented with auto-differentiation libraries, and lead to performance gains in practice.Preprint. Under review.

Biased Gradient Estimate with Drastic Variance Reduction for Meta Reinforcement Learning

“…This is mainly because practical algorithms can only estimate ∇ g V g (θ N ) instead of ∇ g V g (θ ), while the latter is required to estimate J ∞ (θ, g) in an unbiased way. This observation was also hinted at recently in (Tang et al, 2021).…”

“…Prior work in fact constructs the LSF gradient estimate. Since most prior work derive meta-RL gradient estimates based on J ∞ (θ, g) (Foerster et al, 2018;Rothfuss et al, 2018;Liu et al, 2019;Tang et al, 2021), and due to the accidental replacement of θ by θ N , we conclude that they in fact construct variants of the LSF gradient estimate (see comments following Corollary 4.3). In particular, they construct Ĵ such that E[ Ĵ] = E[ ĴN,LSF (θ, g)] but with potentially lower variance.…”

“…Though in theory reducing the variance of ĤN (θ) does not necessarily guarantee improvements, in practice, this seems to be very critical. Variance reduction methods include control variates (Liu et al, 2019), as well as introducing further bias to the estimate of ĤN (θ) (Rothfuss et al, 2018;Tang et al, 2021).…”

“…. Most prior work focus on variance reduction for estimating this function (Foerster et al, 2018;Mao et al, 2019;Rothfuss et al, 2018;Tang et al, 2021).…”

“…Recently, there is a growing interest in establishing performance guarantees for meta-RL algorithms with unbiased gradient estimates (Fallah et al, 2020a). However, since the inception of the field, meta-RL practitioners have almost always implemented biased gradient estimates (Finn et al, 2017;Al-Shedivat et al, 2017;Rothfuss et al, 2018;Liu et al, 2019;Tang et al, 2021). It is natural to ask: why are unbiased gradient estimates potentially undesirable in practice, and what do we gain by introducing bias into gradient estimates?…”