Improving control through subsumption in the EvoTanks domain

IEEE Trans. Comput. Intell. AI Games

2016

Ms. Pac-Man is a challenging video game in which multiple modes of behavior are required: Ms. Pac-Man must escape ghosts when they are threats and catch them when they are edible, in addition to eating all pills in each level. Past approaches to learning behavior in Ms. Pac-Man have treated the game as a single task to be learned using monolithic policy representations. In contrast, this paper uses a framework called Modular Multi-objective NEAT (MM-NEAT) to evolve modular neural networks. Each module defines a separate behavior. The modules are used at different times according to a policy that can be human-designed (i.e. Multitask) or discovered automatically by evolution. The appropriate number of modules can be fixed or discovered using a genetic operator called Module Mutation. Several versions of Module Mutation are evaluated in this paper. Both fixed modular networks and Module Mutation networks outperform monolithic networks and Multitask networks. Interestingly, the best networks dedicate modules to critical behaviors (such as escaping when surrounded after luring ghosts near a power pill) that do not follow the customary division of the game into chasing edible and escaping threat ghosts. The results demonstrate that MM-NEAT can discover interesting and effective behavior for agents in challenging games.

Section: Related Workmentioning

confidence: 99%

Discovering Multimodal Behavior in Ms. Pac-Man Through Evolution of Modular Neural Networks

IEEE Trans. Comput. Intell. AI Games

2016

“…Such hierarchies can be hand-designed (Brooks, 1986) or learned piece by piece. For example, Togelius's (2004) evolved subsumption architecture was used in EvoTanks (Thompson et al, 2009) andUnreal Tournament (van Hoorn et al, 2009), and Stone and Veloso's (2000) Layered Learning was applied to simulated RoboCup Soccer. Recently, Lessin et al (2013) used a human-designed hierarchical syllabus to evolve complex behavior for virtual creatures.…”

Section: Separately Learned Controllersmentioning

confidence: 99%

Solving Multiple Isolated, Interleaved, and Blended Tasks through Modular Neuroevolution

Evolutionary Computation

2016

Many challenging sequential decision-making problems require agents to master multiple tasks. For instance, game agents may need to gather resources, attack opponents, and defend against attacks. Learning algorithms can thus benefit from having separate policies for these tasks, and from knowing when each one is appropriate. How well this approach works depends on how tightly coupled the tasks are. Three cases are identified: Isolated tasks have distinct semantics and do not interact, interleaved tasks have distinct semantics but do interact, and blended tasks have regions where semantics from multiple tasks overlap. Learning across multiple tasks is studied in this article with Modular Multiobjective NEAT, a neuroevolution framework applied to three variants of the challenging Ms. Pac-Man video game. In the standard blended version of the game, a surprising, highly effective machine-discovered task division surpasses human-specified divisions, achieving the best scores to date in this game. In isolated and interleaved versions of the game, human-specified task divisions are also successful, though the best scores are surprisingly still achieved by machine discovery. Modular neuroevolution is thus shown to be capable of finding useful, unexpected task divisions better than those apparent to a human designer.

“…Such hierarchies can be hand-designed [5] or learned. For example, Togelius's [31] evolved subsumption architecture was used in EvoTanks [30] and Unreal Tournament [32], and Stone's [27] Layered Learning was applied to RoboCup Soccer. Recently, Lessin et al [18] used a human-designed hierarchical syllabus to evolve complex behavior for virtual creatures.…”

Section: Related Workmentioning

confidence: 99%

Solving Interleaved and Blended Sequential Decision-Making Problems through Modular Neuroevolution

Proceedings of the 2015 Annual Conference on Genetic and Evolutionary Computation

2015

Many challenging sequential decision-making problems require agents to master multiple tasks, such as defense and offense in many games. Learning algorithms thus benefit from having separate policies for these tasks, and from knowing when each one is appropriate. How well the methods work depends on the nature of the tasks: Interleaved tasks are disjoint and have different semantics, whereas blended tasks have regions where semantics from different tasks overlap. While many methods work well in interleaved tasks, blended tasks are difficult for methods with strict, human-specified task divisions, such as Multitask Learning. In such problems, task divisions should be discovered automatically. To demonstrate the power of this approach, the MM-NEAT neuroevolution framework is applied in this paper to two variants of the challenging video game of Ms. Pac-Man. In the simplified interleaved version of the game, the results demonstrate when and why such machine-discovered task divisions are useful. In the standard blended version of the game, a surprising, highly effective machine-discovered task division surpasses human-specified divisions, achieving the best scores to date in this game. Modular neuroevolution is thus a promising technique for discovering multimodal behavior for challenging real-world tasks.