Evolving Soccer Keepaway Players Through Task Decomposition

Abstract

Abstract. Complex control tasks can often be solved by decomposing them into hierarchies of manageable subtasks. Such decompositions require designers to decide how much human knowledge should be used to help learn the resulting components. On one hand, encoding human knowledge requires manual effort and may incorrectly constrain the learner's hypothesis space or guide it away from the best solutions. On the other hand, it may make learning easier and enable the learner to tackle more complex tasks. This article examines the impact of this trade-off in tasks of varying difficulty. A space laid out by two dimensions is explored: (1) how much human assistance is given and (2) how difficult the task is. In particular, the neuroevolution learning algorithm is enhanced with three different methods for learning the components that result from a task decomposition. The first method, coevolution, is mostly unassisted by human knowledge. The second method, layered learning, is highly assisted. The third method, concurrent layered learning, is a novel combination of the first two that attempts to exploit human knowledge while retaining some of coevolution's flexibility. Detailed empirical results are presented comparing and contrasting these three approaches on two versions of a complex task, namely robot soccer keepaway, that differ in difficulty of learning. These results confirm that, given a suitable task decomposition, neuroevolution can master difficult tasks. Furthermore, they demonstrate that the appropriate level of human assistance depends critically on the difficulty of the problem.

Deep learning in neural networks: An overview

“…In particular, gradient-based subgoal discovery with FNNs or RNNs decomposes RL tasks into subtasks for RL submodules (Schmidhuber, 1991b;Schmidhuber and Wahnsiedler, 1992). Numerous alternative HRL techniques have been proposed (e.g., Ring, 1991Ring, , 1994Jameson, 1991;Tenenberg et al, 1993;Weiss, 1994;Moore and Atkeson, 1995;Precup et al, 1998;Dietterich, 2000b;Menache et al, 2002;Doya et al, 2002;Ghavamzadeh and Mahadevan, 2003;Barto and Mahadevan, 2003;Samejima et al, 2003;Bakker and Schmidhuber, 2004;Whiteson et al, 2005;Simsek and Barto, 2008). While HRL frameworks such as Feudal RL (Dayan and Hinton, 1993) and options (Sutton et al, 1999b;Barto et al, 2004;Singh et al, 2005) do not directly address the problem of automatic subgoal discovery, HQ-Learning (Wiering and Schmidhuber, 1998a) automatically decomposes POMDPs (Sec.…”

Evolving explicit opponent models in game playing

“…The probable locations of the opponent team members were modeled. Further, Whiteson et al [15] worked on RoboCup Keepaway, where three players must keep the ball away from a fourth player. Implicit opponent models are needed in order to avoid the fourth player.…”

Comparing evolutionary and temporal difference methods in a reinforcement learning domain

“…Keepaway is an appealing platform for empirical comparisons because the performance of TD methods has already been established in previous studies [8,23]. While GAs have been applied to variations of Keepaway [7,26], they have never, to our knowledge, been applied to the task's benchmark version.…”

“…There are a wide variety of both GAs and TD methods in use today but in order to compare these different approaches empirically we must focus on specific instantiations. We use Sarsa and NEAT as representative methods because of their empirical success in the benchmark Keepaway task [23] or variations thereof [26].…”