Transfer of evolved pattern-based heuristics in games

Abstract

Learning is key to achieving human-level intelligence. Transferring knowledge that is learned on one task to another one speeds up learning in the target task by exploiting the relevant prior knowledge. As a test case, this study introduces a method to transfer local pattern-based heuristics from a simple board game to a more complex one. The patterns are generated by compositional pattern producing networks (CPPNs), which are evolved with the NEAT neuroevolution method. Results show that transfer improves both final performance and the total learning time, compared to evolving patterns for the target game from scratch. Pattern-based transfer is therefore a promising approach to scaling up game players toward humanlevel.

Evolutionary Policy Transfer and Search Methods for Boosting Behavior Quality: RoboCup Keep-Away Case Study

“…The efficacy of this policy transfer method was supported by improved task performance on target tasks given further behavior evolution. This method was supported by prior work (Bahceci and Miikkulainen, 2008) that evolved behaviors of computer board-game playing agents, where indirectly encoded representations of evolved behaviors facilitated effective agent behavior transfer between games of increasing complexity (board size).…”

“…HyperNEAT was selected as this study's indirect encoding neuroevolution method since previous research indicated that transferring the connectivity patterns (Gauci and Stanley, 2008) of evolved behaviors is an effective way for facilitating transfer learning in multiagent tasks (Bahceci and Miikkulainen, 2008;Verbancsics and Stanley, 2010). That is, HyperNEAT evolved multiagent policies can be effectively transferred to increasingly complex tasks (Stone et al, 2006a) without further adaptation (Verbancsics and Stanley, 2010) and that transferred behaviors often yield comparable task performance to specially designed learning algorithms (Stone et al, 2006b).…”

“…The parameter k is specified by the experimenter to represent the number of nearest neighbors, where k = 15 has been widely used in novelty search experiments (Gomes et al, 2015). Some researchers have used k = 20 (Liapis et al, 2015) and k in the range of Bahceci and Miikkulainen (2008), Gomes et al (2015) though it is unclear if values were derived experimentally. Gomes et al (2015) discovered that the choice of k value depends on the type of novelty archive used and that k = 15 yielded relatively good performance across all tested archive types.…”

Transferring Evolved Reservoir Features in Reinforcement Learning Tasks

“…The transfer of neural networks has been studied in [15] and [1]. More specifically, Taylor et al [15] proposed a method, named Transfer via inter-task mappings for Policy Search Reinforcement Learning (TVITM-PS), that initializes the weights of the target task, using the learned weights of the source task, by utilizing mapping functions.…”

“…Additionally, we have tested a method that tries to take advantage of the fully, random, connectivity of ESNs and does not require the use of mappings, with promising results. Finally, in [1] Bahceci and Miikkulainen introduce a method that transfers pattern-based heuristic in games. A population of evolved neural networks (which represent the patterns) to a target task as the starting population.…”