Using Advice to Transfer Knowledge Acquired in One Reinforcement Learning Task to Another

Abstract

Abstract. We present a method for transferring knowledge learned in one task to a related task. Our problem solvers employ reinforcement learning to acquire a model for one task. We then transform that learned model into advice for a new task. A human teacher provides a mapping from the old task to the new task to guide this knowledge transfer. Advice is incorporated into our problem solver using a knowledge-based support vector regression method that we previously developed. This advice-taking approach allows the problem solver to refine or even discard the transferred knowledge based on its subsequent experiences. We empirically demonstrate the effectiveness of our approach with two games from the RoboCup soccer simulator: KeepAway and BreakAway. Our results demonstrate that a problem solver learning to play BreakAway using advice extracted from KeepAway outperforms a problem solver learning without the benefit of such advice.

Learning potential functions and their representations for multi-task reinforcement learning

“…This approach also makes it easier to override the bias of the cross-task function, which is important when that bias proves incorrect [9]. The cross-task function can be represented as, e.g., advice rules [81], or a shaping function [34,53], the approach we focus on in this article.…”

Knowledge Intensive Learning: Combining Qualitative Constraints with Causal Independence for Parameter Learning in Probabilistic Models

“…However in order to reach the optimal performance when applied to diverse problems, machine learning technology need large amounts of training data and high cost of manually designing the hypothesis space of the learning system. Advice giving for Machine learning is an exciting research direction that can fill the gap between using only examples that will rely on a large number of examples in complex domains and expert systems that require the model to be exactly specified by the expert [15,16,17,18,9,19,20,21]. If only a small amount of data is available, then the prediction accuracy based on the knowledge learned from the data would be inevitably limited at a low level.…”

Probabilistic Policy Reuse for inter-task transfer learning

“…For instance, this mapping can be done by an expert who decides the correspondence among states and actions [14,10]. That mapping can also be learned by observing a mentor [16].…”

“…The method is based on a mapping among policies. Such mappings between state and action spaces are required in other approaches for transfer learning [9,10]. Given that we only need to transfer policies for policy reuse, interestingly and as shown in this paper, the amount of knowledge required by our mapping is much smaller than in methods based on the transfer of the value function.…”

“…We apply the transfer learning using Policy Reuse to the Keepaway framework [4], which favorably compares with other transfer learning methods that have been recently applied [14,10,15] to this same domain.…”