Solving Interleaved and Blended Sequential Decision-Making Problems through Modular Neuroevolution

Abstract

Many challenging sequential decision-making problems require agents to master multiple tasks, such as defense and offense in many games. Learning algorithms thus benefit from having separate policies for these tasks, and from knowing when each one is appropriate. How well the methods work depends on the nature of the tasks: Interleaved tasks are disjoint and have different semantics, whereas blended tasks have regions where semantics from different tasks overlap. While many methods work well in interleaved tasks, blended tasks are difficult for methods with strict, human-specified task divisions, such as Multitask Learning. In such problems, task divisions should be discovered automatically. To demonstrate the power of this approach, the MM-NEAT neuroevolution framework is applied in this paper to two variants of the challenging video game of Ms. Pac-Man. In the simplified interleaved version of the game, the results demonstrate when and why such machine-discovered task divisions are useful. In the standard blended version of the game, a surprising, highly effective machine-discovered task division surpasses human-specified divisions, achieving the best scores to date in this game. Modular neuroevolution is thus a promising technique for discovering multimodal behavior for challenging real-world tasks.

Using Indirect Encoding of Multiple Brains to Produce Multimodal Behavior

“…However, because effective task divisions were available, preference neurons are never significantly better than SPG, and are inferior to MT. This result contrasts with previous results in Ms. Pac-Man using the direct encoding MM-NEAT [4], [17]. It is possible that preference neurons are less effective when combined with HyperNEAT than with directly encoded neural networks, but it is more likely that the increased complexity of Ms. Pac-Man (compared to the domains of this paper) is what allowed preference neurons to shine.…”

“…For these reasons, modular ANNs are an active area of research [13], [14], [15]. Most multimodal approaches either implement evolutionary mechanisms that encourage modularity [13], [14] or explicitly divide ANNs into modules that can specialize to different tasks [16], [3], [4], [17].…”

Solving Multiple Isolated, Interleaved, and Blended Tasks through Modular Neuroevolution

“…Most of the modular architectures described above were applied to the original, blended version of Ms. Pac-Man (Schrum and Miikkulainen, 2014), in which a mixture of edible and threat ghosts can be present at the same time. Later, an interleaved version of the domain was introduced (Schrum and Miikkulainen, 2015), in which such mixtures never occur. In this article, a new isolated version of the domain is introduced in which ghost eating is completely isolated from pill eating.…”