Metatrace Actor-Critic: Online Step-Size Tuning by Meta-gradient Descent for Reinforcement Learning Control

Abstract

Reinforcement learning (RL) has had many successes in both "deep" and "shallow" settings. In both cases, significant hyperparameter tuning is often required to achieve good performance. Furthermore, when nonlinear function approximation is used, non-stationarity in the state representation can lead to learning instability. A variety of techniques exist to combat this -most notably large experience replay buffers or the use of multiple parallel actors. These techniques come at the cost of moving away from the online RL problem as it is traditionally formulated (i.e., a single agent learning online without maintaining a large database of training examples). Meta-learning can potentially help with both these issues by tuning hyperparameters online and allowing the algorithm to more robustly adjust to non-stationarity in a problem. This paper applies meta-gradient descent to derive a set of step-size tuning algorithms specifically for online RL control with eligibility traces. Our novel technique, Metatrace, makes use of an eligibility trace analogous to methods like T D(λ). We explore tuning both a single scalar step-size and a separate step-size for each learned parameter. We evaluate Metatrace first for control with linear function approximation in the classic mountain car problem and then in a noisy, non-stationary version. Finally, we apply Metatrace for control with nonlinear function approximation in 5 games in the Arcade Learning Environment where we explore how it impacts learning speed and robustness to initial step-size choice. Results show that the meta-step-size parameter of Metatrace is easy to set, Metatrace can speed learning, and Metatrace can allow an RL algorithm to deal with non-stationarity in the learning task.

Meta-Learning in Neural Networks: A Survey

“…Several previously discussed meta-representations have been explored in RL including learning the initial conditions [19], [169], hyperparameters [169], [173], step directions [76] and step sizes [171]. These enable gradient-based learning of a neural policy with fewer environmental interactions and training fast convolutional [38] or recurrent [23], [112] blackbox models to synthesize a policy by embedding the enviroment experience.…”

“…[91] [16], [76], [103], [104], [170] PSD [78]. [90] Hyperparam HyperRep [20], HyperOpt [66], LHML [68] MetaTrace [171]. [172] [169] [173] Feed-Forward model SNAIL [38], CNAP [107].…”

“…Online meta-RL A significant fraction of meta-RL studies addressed the single-task setting, where the metaknowledge such as loss [117], [179], reward [173], [180], hyperparameters [171], [172], or exploration strategy [181] are trained online together with the base policy while learning a single task. These methods thus do not require task families and provide a direct improvement to their respective base learners' performance.…”

Learning to Accelerate by the Methods of Step-size Planning

“…In reinforcement leanring, Kearney et al (2018) extended IDBD to TD methods for stationary and non-stationary prediction tasks, which showed better results than TD methods with the constant step-size and scalar step-size adaptation. Metatrace (Young et al, 2018) adapted the step-size of the actor-critic algorithm with eligibility trace for both cases of the scalar step-size and the component step-sizes, which can accelerate the learning process of the actor-critic algorithm.…”

Balancing Constraints and Rewards with Meta-Gradient D4PG

“…Meta-gradients have been previously used to learn intrinsic rewards for policy gradient (Zheng et al, 2018) and auxiliary tasks (Veeriah et al, 2019). Meta-gradients have also been used to adapt optimizer parameters (Young et al, 2018;Franceschi et al, 2017). In our setup, we consider the continuous control setting, provide the first implementation of metagradients for an algorithm that uses an experience replay, and focus on adapting meta-parameters that encourage soft constraint satisfaction while maximizing expected return.…”