Learning potential functions and their representations for multi-task reinforcement learning

Snel, Matthijs; Whiteson, Shimon

doi:10.1007/s10458-013-9235-z

Cited by 34 publications

(12 citation statements)

References 54 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We provide insights into optimal reward shaping, and propose a novel meta-learning framework to automatically learn such reward shaping to apply on newly sampled tasks. Theoretical analysis and extensive experiments establish us as the state-of-the-art in learning task-distribution reward shaping, outperforming previous such works ( Konidaris and Barto 2006;Snel and Whiteson 2014). We further show that our method outperforms learning intrinsic rewards (Yang et al 2019;Zheng et al 2020), outperforms Rainbow (Hessel et al 2018) in complex pixel-based CoinRun games, and is also better than hand-designed reward shaping on grid mazes.…”

supporting

confidence: 57%

Learning Task-Distribution Reward Shaping with Meta-Learning

Zou

Ren

Yan

et al. 2021

AAAI

View full text Add to dashboard Cite

Reward shaping is one of the most effective methods to tackle the crucial yet challenging problem of credit assignment and accelerate Reinforcement Learning. However, designing shaping functions usually requires rich expert knowledge and hand-engineering, and the difficulties are further exacerbated given multiple tasks to solve. In this paper, we consider reward shaping on a distribution of tasks that share state spaces but not necessarily action spaces. We provide insights into optimal reward shaping, and propose a novel meta-learning framework to automatically learn such reward shaping to apply on newly sampled tasks. Theoretical analysis and extensive experiments establish us as the state-of-the-art in learning task-distribution reward shaping, outperforming previous such works (Konidaris and Barto 2006; Snel and Whiteson 2014). We further show that our method outperforms learning intrinsic rewards (Yang et al. 2019; Zheng et al. 2020), outperforms Rainbow (Hessel et al. 2018) in complex pixel-based CoinRun games, and is also better than hand-designed reward shaping on grids. While the goal of this paper is to learn reward shaping rather than to propose new general meta-learning algorithms as PEARL (Rakelly et al. 2019) or MQL (Fakoor et al. 2020), our framework based on MAML (Finn, Abbeel, and Levine 2017) also outperforms PEARL / MQL, and could combine with them for further improvement.

show abstract

supporting

confidence: 57%

Learning Task-Distribution Reward Shaping with Meta-Learning

Zou

Ren

Yan

et al. 2021

AAAI

View full text Add to dashboard Cite

show abstract

“…However, potential functions will typically need to be pre-specified. This has restricted the use of PBRS to tabular/ low-dimensional state spaces [15,38]. The cycling problem (repeatedly visiting certain states) mention in Section 4.1 can be resolved by PBRS.…”

Section: Related Workmentioning

confidence: 99%

FRESH: Interactive Reward Shaping in High-Dimensional State Spaces using Human Feedback

Xiao,

Lu,

Ramasubramanian

et al. 2020

Preprint

View full text Add to dashboard Cite

Reinforcement learning has been successful in training autonomous agents to accomplish goals in complex environments. Although this has been adapted to multiple settings, including robotics and computer games, human players often find it easier to obtain higher rewards in some environments than reinforcement learning algorithms. This is especially true of high-dimensional state spaces where the reward obtained by the agent is sparse or extremely delayed. In this paper, we seek to effectively integrate feedback signals supplied by a human operator with deep reinforcement learning algorithms in high-dimensional state spaces. We call this FRESH (Feedback-based REward SHaping). During training, a human operator is presented with trajectories from a replay buffer and then provides feedback on states and actions in the trajectory. In order to generalize feedback signals provided by the human operator to previously unseen states and actions at test-time, we use a feedback neural network. We use an ensemble of neural networks with a shared network architecture to represent model uncertainty and the confidence of the neural network in its output. The output of the feedback neural network is converted to a shaping reward that is augmented to the reward provided by the environment. We evaluate our approach on the Bowling and Skiing Atari games in the arcade learning environment. Although human experts have been able to achieve high scores in these environments, state-of-the-art deep learning algorithms perform poorly. We observe that FRESH is able to achieve much higher scores than state-of-the-art deep learning algorithms in both environments. FRESH also achieves a 21.4% higher score than a human expert in Bowling and does as well as a human expert in Skiing.

show abstract

“…The multi-task community has roughly separated learnable knowledge into two categories [44]. Task relevant knowledge pertains to a particular task [23,51]; meanwhile, domain relevant knowledge is common across all tasks [6,14,25].…”

Section: Multi-task Learningmentioning

confidence: 99%

Learning to Play against Any Mixture of Opponents

Smith¹,

Anthony²,

Wang³

et al. 2020

Preprint

View full text Add to dashboard Cite

Intuitively, experience playing against one mixture of opponents in a given domain should be relevant for a different mixture in the same domain. We propose a transfer learning method, Q-Mixing, that starts by learning Q-values against each pure-strategy opponent. Then a Q-value for any distribution of opponent strategies is approximated by appropriately averaging the separately learned Q-values. From these components, we construct policies against all opponent mixtures without any further training. We empirically validate Q-Mixing in two environments: a simple grid-world soccer environment, and a complicated cyber-security game. We find that Q-Mixing is able to successfully transfer knowledge across any mixture of opponents. We next consider the use of observations during play to update the believed distribution of opponents. We introduce an opponent classifier-trained in parallel to Q-learning, using the same data-and use the classifier results to refine the mixing of Q-values. We find that Q-Mixing augmented with the opponent classifier function performs comparably, and with lower variance, than training directly against a mixed-strategy opponent.

show abstract

Learning potential functions and their representations for multi-task reinforcement learning

Cited by 34 publications

References 54 publications

Learning Task-Distribution Reward Shaping with Meta-Learning

Learning Task-Distribution Reward Shaping with Meta-Learning

FRESH: Interactive Reward Shaping in High-Dimensional State Spaces using Human Feedback

Learning to Play against Any Mixture of Opponents

Contact Info

Product

Resources

About