Tuning-free step-size adaptation

Mahmood, A. Rupam; Sutton, Richard S.; Degris, Thomas; Pilarski, Patrick M.

doi:10.1109/icassp.2012.6288330

Cited by 42 publications

(49 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We demonstrate that Scalar Metatrace improves robustness to initial step-size choice in a standard RL domain, while Mixed Metatrace facilitates learning in an RL problem with nonstationary state representation. The latter result extends results of [15] and [8] from the SL case. Reasoning that such non-stationarity in the state representation is an inherent feature of NN function approximation, we also test the method for training a neural network online for several games in the ALE.…”

Section: Resultssupporting

confidence: 90%

“…This entropy regularized extension to the basic algorithm is used in lines 7, 10, and 14 of algorithm 1. Algorithm 1 also incorporates a normalization technique, analogous to that used in [8], that we will now discuss.…”

Section: Scalar Metatrace For Ac (λ)mentioning

confidence: 99%

“…A reasonable way to proceed then would be to simply choose our shared (scalar or vector) α to ensure that the maximum effective-step size for any state contributing to the current trace is still less than one to avoid overshooting. We maintain a running maximum of effective step-sizes over states in the trace, similar to the running maximum of β updates used in [8], and multiplicatively decaying our αs on each time-step by the amount this exceeds one. This procedure is shown on lines 11, 12 and 13 of algorithm 1.…”

Section: Scalar Metatrace For Ac (λ)mentioning

confidence: 99%

“…Other options for normalizing the step-size may be worth considering as well. We will depart from [8] somewhat by choosing µ itself as the tracking parameter for v rather than 1…”

Section: Scalar Metatrace For Ac (λ)mentioning

confidence: 99%

See 3 more Smart Citations

Metatrace Actor-Critic: Online Step-Size Tuning by Meta-gradient Descent for Reinforcement Learning Control

Young

Wang²,

Taylor³

2019

Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence

View full text Add to dashboard Cite

Reinforcement learning (RL) has had many successes in both "deep" and "shallow" settings. In both cases, significant hyperparameter tuning is often required to achieve good performance. Furthermore, when nonlinear function approximation is used, non-stationarity in the state representation can lead to learning instability. A variety of techniques exist to combat this -most notably large experience replay buffers or the use of multiple parallel actors. These techniques come at the cost of moving away from the online RL problem as it is traditionally formulated (i.e., a single agent learning online without maintaining a large database of training examples). Meta-learning can potentially help with both these issues by tuning hyperparameters online and allowing the algorithm to more robustly adjust to non-stationarity in a problem. This paper applies meta-gradient descent to derive a set of step-size tuning algorithms specifically for online RL control with eligibility traces. Our novel technique, Metatrace, makes use of an eligibility trace analogous to methods like T D(λ). We explore tuning both a single scalar step-size and a separate step-size for each learned parameter. We evaluate Metatrace first for control with linear function approximation in the classic mountain car problem and then in a noisy, non-stationary version. Finally, we apply Metatrace for control with nonlinear function approximation in 5 games in the Arcade Learning Environment where we explore how it impacts learning speed and robustness to initial step-size choice. Results show that the meta-step-size parameter of Metatrace is easy to set, Metatrace can speed learning, and Metatrace can allow an RL algorithm to deal with non-stationarity in the learning task.

show abstract

Section: Resultssupporting

confidence: 90%

Section: Scalar Metatrace For Ac (λ)mentioning

confidence: 99%

Section: Scalar Metatrace For Ac (λ)mentioning

confidence: 99%

“…Other options for normalizing the step-size may be worth considering as well. We will depart from [8] somewhat by choosing µ itself as the tracking parameter for v rather than 1…”

Section: Scalar Metatrace For Ac (λ)mentioning

confidence: 99%

See 2 more Smart Citations

Metatrace Actor-Critic: Online Step-Size Tuning by Meta-gradient Descent for Reinforcement Learning Control

Young

Wang²,

Taylor³

2019

Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence

View full text Add to dashboard Cite

show abstract

“…Recently, Mahmood and Sutton [17], [18] proposed with Autostep an extension to IDBD which has much less dependence on the meta-step-size parameter than IDBD. In the same year, Dabney and Barto [19] developed another adaptive step-size method for temporal difference learning, which is based on the estimation of upper and lower bounds.…”

Section: Related Workmentioning

confidence: 99%

Online Adaptable Learning Rates for the Game Connect-4

Bagheri

Thill

Koch

et al. 2016

IEEE Trans. Comput. Intell. AI Games

View full text Add to dashboard Cite

Learning board games by self-play has a long tradition in computational intelligence for games. Based on Tesauro's seminal success with TD-Gammon in 1994, many successful agents use temporal difference learning today. But in order to be successful with temporal difference learning on game tasks, often a careful selection of features and a large number of training games is necessary. Even for board games of moderate complexity like Connect-4, we found in previous work that a very rich initial feature set and several millions of game plays are required. In this work we investigate different approaches of online-adaptable learning rates like Incremental Delta Bar Delta (IDBD) or Temporal Coherence Learning (TCL) whether they have the potential to speed up learning for such a complex task. We propose a new variant of TCL with geometric step size changes. We compare those algorithms with several other state-of-the-art learning rate adaptation algorithms and perform a case study on the sensitivity with respect to their meta parameters. We show that in this set of learning algorithms those with geometric step size changes outperform those other algorithms with constant step size changes. Algorithms with nonlinear output functions are slightly better than linear ones. Algorithms with geometric step size changes learn faster by a factor of 4 as compared to previously published results on the task Connect-4.

show abstract

Learning with Random Learning Rates

Blier

Wolinski

Ollivier³

2020

Lecture Notes in Computer Science

View full text Add to dashboard Cite

In neural networks, the learning rate of the gradient descent strongly affects performance. This prevents reliable out-of-the-box training of a model on a new problem. We propose the All Learning Rates At Once (Alrao) algorithm: each unit or feature in the network gets its own learning rate sampled from a random distribution spanning several orders of magnitude, in the hope that enough units will get a close-to-optimal learning rate. Perhaps surprisingly, stochastic gradient descent (SGD) with Alrao performs close to SGD with an optimally tuned learning rate, for various network architectures and problems. In our experiments, all Alrao runs were able to learn well without any tuning.

show abstract

Tuning-free step-size adaptation

Cited by 42 publications

References 7 publications

Metatrace Actor-Critic: Online Step-Size Tuning by Meta-gradient Descent for Reinforcement Learning Control

Metatrace Actor-Critic: Online Step-Size Tuning by Meta-gradient Descent for Reinforcement Learning Control

Online Adaptable Learning Rates for the Game Connect-4

Learning with Random Learning Rates

Contact Info

Product

Resources

About