Skill Acquisition Via Transfer Learning and Advice Taking

Abstract

Abstract. We describe a reinforcement learning system that transfers skills from a previously learned source task to a related target task. The system uses inductive logic programming to analyze experience in the source task, and transfers rules for when to take actions. The target task learner accepts these rules through an advice-taking algorithm, which allows learners to benefit from outside guidance that may be imperfect. Our system accepts a human-provided mapping, which specifies the similarities between the source and target tasks and may also include advice about the differences between them. Using three tasks in the RoboCup simulated soccer domain, we demonstrate that this system can speed up reinforcement learning substantially.

Relational Macros for Transfer in Reinforcement Learning

“…Our previous work includes a method called skill transfer [20]. In skill transfer, we learn rules with ILP that indicate when the agent chooses to take a single source-task action.…”

“…Note that one final step might be necessary if the actions and features in the source and target tasks are not identically named: a mapping from source-task names to target-task names, as in Torrey et al [20,21]. Our approach does not even require the tasks to be completely isomorphic, because we can set the Aleph language restrictions so that only source-task elements that have corresponding target-task elements appear in the macro.…”

“…In our own recent work [20], we introduce an approach that transfers skills using inductive logic programming (ILP), where a skill is a type of action that the RL agent uses. In this paper, we extend that approach by transferring strategies, which are action plans that may require composing several skills.…”

Transfer Learning in Multi-Agent Reinforcement Learning Domains

“…More flexible methods allow the tasks to differ in the state and action spaces (with different variables in both sets) and also in the reward and transition functions [13,12]. These methods use inter-task mappings in order to relate the source and target tasks.…”

Transfer Learning in Reinforcement Learning Problems Through Partial Policy Recycling

“…The advice is incorporated into the new task by adding the information about Q-values as soft-constraints to the linear optimization problem that approximates the Q-function for the next task. In follow up work [12], the advice is generated from the original Q-function automatically through the use of a relational rule learner and extended with user-defined advice. Although this approach does incorporate knowledge about the Q-values of the first task into the construction of the Q-function of the second task, we still feel that a lot of (possibly useful) knowledge about the structure of the Q-function is lost.…”