Learning to reinforcement learn

Wang, Jane X.; Kurth‐Nelson, Zeb; Tirumala, Dhruva; Soyer, Hubert; Leibo, Joel Z.; Munos, Rémi; Blundell, Charles; Kumaran, Dharshan; Botvinick, Matt

doi:10.48550/arxiv.1611.05763

Cited by 165 publications

(250 citation statements)

References 26 publications

(49 reference statements)

Supporting

Mentioning

232

Contrasting

Order By: Relevance

“…The works in [11], [12] validate a meta-learning framework that can be used in several learning tasks, e.g., it can be applied to both supervised ML (regression and classification) and RL scenarios. Other works propose metalearning for more specific scenarios, i.e., the update rule and selective copy of weights of deep networks [36], [37], [38] and recurrent networks [39], [40], [41]. In this paper, we design FALCON based on meta-learning paradigm to obtain fast and accurate scheduling policies.…”

Section: Learning Concepts In Networkingmentioning

confidence: 99%

FALCON: Fast and Accurate Multipath Scheduling using Offline and Online Learning

Wu¹,

Alay²,

Brunström³

et al. 2022

Preprint

View full text Add to dashboard Cite

Multipath transport protocols enable the concurrent use of different network paths, benefiting a fast and reliable data transmission. The scheduler of a multipath transport protocol determines how to distribute data packets over different paths. Existing multipath schedulers either conform to predefined policies or to online trained policies. The adoption of millimeter wave (mmWave) paths in 5th Generation (5G) networks and Wireless Local Area Networks (WLANs) introduces timevarying network conditions, under which the existing schedulers struggle to achieve fast and accurate adaptation. In this paper, we propose FALCON, a learning-based multipath scheduler that can adapt fast and accurately to time-varying network conditions. FALCON builds on the idea of meta-learning where offline learning is used to create a set of meta-models that represent coarse-grained network conditions, and online learning is used to bootstrap a specific model for the current fine-grained network conditions towards deriving the scheduling policy to deal with such conditions. Using trace-driven emulation experiments, we demonstrate FALCON outperforms the best state-of-the-art scheduler by up to 19.3% and 23.6% in static and mobile networks, respectively. Furthermore, we show FALCON is quite flexible to work with different types of applications such as bulk transfer and web services. Moreover, we observe FALCON has a much faster adaptation time compared to all the other learningbased schedulers, reaching almost an 8-fold speedup compared to the best of them. Finally, we have validated the emulation results in real-world settings illustrating that FALCON adapts well to the dynamicity of real networks, consistently outperforming all other schedulers.

show abstract

Section: Learning Concepts In Networkingmentioning

confidence: 99%

FALCON: Fast and Accurate Multipath Scheduling using Offline and Online Learning

Wu¹,

Alay²,

Brunström³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Meta-RL tries to enable an agent to solve unseen tasks and environments efficiently. To this end, meta-RL methods infer the task over a latent context (Lin et al, 2020;Rakelly et al, 2019), redesign policy neural network architecture (Duan et al, 2016;Wang et al, 2016;Lan et al, 2019), control exploration strategies (Gupta et al, 2018), and augment state or reward (Raileanu et al, 2020;Florensa et al, 2018). These previous methods commonly focus on the family of tasks described by a family of MDPs which all share the same dynamics and environment but differ in the task specified by the reward function (Lin et al, 2020).…”

Section: Multiplementioning

confidence: 99%

System-Agnostic Meta-Learning for MDP-based Dynamic Scheduling via Descriptive Policy

Lee¹

2022

Preprint

View full text Add to dashboard Cite

Dynamic scheduling is an important problem in applications from queuing to wireless networks. It addresses how to choose an item among multiple scheduling items in each timestep to achieve a long-term goal. Conventional approaches for dynamic scheduling find the optimal policy for a given specific system so that the policy from these approaches is usable only for the corresponding system characteristics. Hence, it is hard to use such approaches for a practical system in which system characteristics dynamically change. This paper proposes a novel policy structure for MDP-based dynamic scheduling, a descriptive policy, which has a system-agnostic capability to adapt to unseen system characteristics for an identical task (dynamic scheduling). To this end, the descriptive policy learns a system-agnostic scheduling principle-in a nutshell, "which condition of items should have a higher priority in scheduling". The scheduling principle can be applied to any system so that the descriptive policy learned in one system can be used for another system. Experiments with simple explanatory and realistic application scenarios demonstrate that it enables system-agnostic meta-learning with very little performance degradation compared with the system-specific conventional policies.

show abstract

“…Both the fixed payoff and the mean 𝜇 of the risky arm were drawn from a standard Gaussian distribution at the beginning of an episode, which lasted twenty rounds. To build agents that can trade off exploration versus exploitation, we used memory-based meta-learning [Santoro et al, 2016, Wang et al, 2016, which is known to produce near-optimal bandit players [Mikulik et al, 2020, Ortega et al, 2019.…”

Section: Banditsmentioning

confidence: 99%

Model-Free Risk-Sensitive Reinforcement Learning

Grégoire¹,

Grau-Moya²,

Kunesch³

et al. 2021

Preprint

View full text Add to dashboard Cite

We extend temporal-difference (TD) learning in order to obtain risk-sensitive, model-free reinforcement learning algorithms. This extension can be regarded as modification of the Rescorla-Wagner rule, where the (sigmoidal) stimulus is taken to be either the event of over-or underestimating the TD target. As a result, one obtains a stochastic approximation rule for estimating the free energy from i.i.d. samples generated by a Gaussian distribution with unknown mean and variance. Since the Gaussian free energy is known to be a certainty-equivalent sensitive to the mean and the variance, the learning rule has applications in risk-sensitive decision-making.

show abstract

Learning to reinforcement learn

Cited by 165 publications

References 26 publications

FALCON: Fast and Accurate Multipath Scheduling using Offline and Online Learning

FALCON: Fast and Accurate Multipath Scheduling using Offline and Online Learning

System-Agnostic Meta-Learning for MDP-based Dynamic Scheduling via Descriptive Policy

Model-Free Risk-Sensitive Reinforcement Learning

Contact Info

Product

Resources

About