Transferring task models in Reinforcement Learning agents

Abstract

The main objective of Transfer Learning is to reuse knowledge acquired in a previous learned task, in order to enhance the learning procedure in a new and more complex task. Transfer learning comprises a suitable solution for speeding up the learning procedure in Reinforcement Learning tasks. This work proposes a novel method for transferring models to Reinforcement Learning agents. The models of the transition and reward functions of a source task, will be transferred to a relevant but different target task. The learning algorithm of the target task's agent takes a hybrid approach, implementing both model-free and model-based learning, in order to fully exploit the presence of a source task model. Moreover, a novel method is proposed for transferring models of potential-based, reward shaping functions. The empirical evaluation, of the proposed approaches, demonstrated significant results and performance improvements in the 3D Mountain Car and Server Job Scheduling tasks, by successfully using the models generated from their corresponding source tasks.

Transferring knowledge as heuristics in reinforcement learning: A case-based approach

“…The experiments verified the initial claim that L3 outperforms its non transfer-learning versions, and also state-of-the-art transfer learning algorithms. The results show that L3 outperforms traditional RL algorithms, such as Q-Learning and SARSA(λ), HARL algorithms such as the HAQL and HA-SARSA(λ), and TL algorithms such as Taylor's MASTER (Taylor, 2008) and TiMRLA Value-Addition algorithms (Fachantidis et al, 2013). It is worth pointing out that, in contrast to L3, HAQL and HA-SARSA(λ) presuppose user-defined domain knowledge.…”

“…L3 was compared with other state-of-the-art transfer learning algorithms: L3 was compared with the the TiMRLA Value-Addition algorithm transfer learning algorithm (Fachantidis et al, 2013) and Taylor's MASTER algorithm (Taylor, 2008) on the Mountain Car experiment (Section 5.1). We did not find any competing algorithm in the literature to compare the Humanoid Robot Stabilisation experiment (presented in Section 5.2), which was a domain included in this paper to illustrate the generality of the algorithm proposed.…”

“…In this work, in order to avoid negative mappings, the authors propose a method for selecting the most relevant mappings. A hybrid approach implementing model-free and model-based learning for transferring models of potential-based, reward shaping functions is proposed in (Fachantidis et al, 2013), whereby the transition and reward functions of the source task are obtained by cascade neural networks. A value-function approximation (the Cerebral Model Articulation Controller, CMAC (Albus, 1975)) is used to find the value of a state, given the neighbouring state values.…”

“…For this reason, the algorithms proposed in (Fachantidis et al, 2013) and in (Taylor, 2008) (mentioned above) are used in the present paper for comparison purposes, as described in Section 5.…”

Shaping in reinforcement learning via knowledge transferred from human-demonstrations

“…Fachantidis et al [10] transfer the model of the transition and reward functions of a source task to a relevant but different target task, and then the agent takes a hybrid approach, implementing both model-free and model-based learning. But all the two methods are limited by the need of inter-task mapping.…”