Discovering Multimodal Behavior in Ms. Pac-Man Through Evolution of Modular Neural Networks

Schrum, Jacob; Miikkulainen, Risto

doi:10.1109/tciaig.2015.2390615

Cited by 36 publications

(69 citation statements)

References 43 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The latter make learning harder, but it is important to be able to learn with conflict sensors, since split sensors are not available in all domains. Schrum and Miikkulainen (2016a) go on to demonstrate that evolving modular neural networks is a way to learn multimodal behaviors with conflict sensors. We test our methods on a task with a very explicit task decomposition at the input level, similar to "split sensors", and on tasks with much less obvious mappings from network inputs/outputs to modular decompositions -studying the effect of structural objectives also on tasks closer to real-world scenarios, where we are not always sure to which module a neuron belongs.…”

Section: Techniques For Leveraging Modularity In Neuroevolutionmentioning

confidence: 99%

Guiding Neuroevolution with Structural Objectives

Ellefsen

Huizinga

Tørresen

2020

Evolutionary Computation

View full text Add to dashboard Cite

The structure and performance of neural networks are intimately connected, and by use of evolutionary algorithms, neural network structures optimally adapted to a given task can be explored. Guiding such neuroevolution with additional objectives related to network structure has been shown to improve performance in some cases, especially when modular neural networks are beneficial. However, apart from objectives aiming to make networks more modular, such structural objectives have not been widely explored. We propose two new structural objectives and test their ability to guide evolving neural networks on two problems which can benefit from decomposition into subtasks. The first structural objective guides evolution to align neural networks with a user-recommended decomposition pattern. Intuitively, this should be a powerful guiding target for problems where human users can easily identify a structure. The second structural objective guides evolution towards a population with a high diversity in decomposition patterns. This results in exploration of many different ways to decompose a problem, allowing evolution to find good decompositions faster. Tests on our target problems reveal that both methods perform well on a problem with a very clear and decomposable structure. However, on a problem where the optimal decomposition is less obvious, the structural diversity objective is found to outcompete other structural objectives -and this technique can even increase performance on problems without any decomposable structure at all.

show abstract

Section: Techniques For Leveraging Modularity In Neuroevolutionmentioning

confidence: 99%

Guiding Neuroevolution with Structural Objectives

Ellefsen

Huizinga

Tørresen

2020

Evolutionary Computation

View full text Add to dashboard Cite

show abstract

“…Each idea is inspired by the direct-encoded Modular Multiobjective NEAT [3]: (1) The network structure from Multitask Learning is adapted to create multitask CPPNs (Figure 1c). In this case, a human must still specify when each policy is used.…”

Section: Background and Extensionsmentioning

confidence: 99%

“…(2) Preference neurons [3] are added that allow evolution to discover when an agent should use each brain ( Figure 1d). In this way, a different brain can be active on each time step, allowing evolution to autonomously discover an effective task division.…”

Section: Background and Extensionsmentioning

confidence: 99%

See 1 more Smart Citation

Automatic Evolution of Multimodal Behavior with Multi-Brain HyperNEAT

Schrum

Lehman

Risi

2016

Proceedings of the 2016 on Genetic and Evolutionary Computation Conference Companion

Self Cite

View full text Add to dashboard Cite

An important challenge in neuroevolution is to evolve multimodal behavior. Indirect network encodings can potentially answer this challenge. Yet in practice, indirect encodings do not yield effective multimodal controllers. This paper introduces novel multimodal extensions to HyperNEAT, a popular indirect encoding. A previous multimodal approach called situational policy geometry assumes that multiple brains benefit from being embedded within an explicit geometric space. However, this paper introduces HyperNEAT extensions for evolving many brains without assuming geometric relationships between them. The resulting Multi-Brain HyperNEAT can exploit human-specified task divisions, or can automatically discover when brains should be used, and how many to use. Experiments show that multi-brain approaches are more effective than HyperNEAT without multimodal extensions, and that brains without a geometric relation to each other are superior.

show abstract

“…However, after activating certain power-ups, the ghosts become vulnerable for a brief period of time. The agent can consume these ghosts for a score boost.The switch in ghost dynamics necessitates a change in the game-play strategy, since multiple distinct modes of behavior are required under different conditions [4,5]. Despite the need for multi-modal behaviors, conventional reinforcement-learning approaches have focused on constructing monolithic policies.…”

mentioning

confidence: 99%

Analysis of Agent Expertise in Ms. Pac-Man Using Value-of-Information-Based Policies

Sledge

Prı́ncipe

2019

IEEE Trans. Games

View full text Add to dashboard Cite

Conventional reinforcement learning methods for Markov decision processes rely on weakly-guided, stochastic searches to drive the learning process. It can therefore be difficult to predict what agent behaviors might emerge. In this paper, we consider an information-theoretic cost function for performing constrained stochastic searches that promote the formation of risk-averse to risk-favoring behaviors. This cost function is the value of information, which provides the optimal trade-off between the expected return of a policy and the policy's complexity; policy complexity is measured by number of bits and controlled by a single hyperparameter on the cost function. As the policy complexity is reduced, the agents will increasingly eschew risky actions. This reduces the potential for high accrued rewards. As the policy complexity increases, the agents will take actions, regardless of the risk, that can raise the long-term rewards. The obtainable reward depends on a single, tunable hyperparameter that regulates the degree of policy complexity.We evaluate the performance of value-of-information-based policies on a stochastic version of Ms. Pac-Man. A major component of this paper is the demonstration that ranges of policy complexity values yield different game-play styles and explaining why this occurs. We also show that our reinforcementlearning search mechanism is more efficient than the others we utilize. This result implies that the value of information theory is appropriate for framing the exploitation-exploration trade-off in reinforcement learning.Index Terms-Value of information, constrained search, reinforcement learning, information theory Isaac J. Sledge is with the 2 objective of the agent is to clear the environment of pellets while navigating around the ghosts. However, after activating certain power-ups, the ghosts become vulnerable for a brief period of time. The agent can consume these ghosts for a score boost.The switch in ghost dynamics necessitates a change in the game-play strategy, since multiple distinct modes of behavior are required under different conditions [4,5]. Despite the need for multi-modal behaviors, conventional reinforcement-learning approaches have focused on constructing monolithic policies. Such policies would implement the same agent behaviors regardless of the vulnerability of the ghosts. Although it is possible to represent multimodal behavior with these policies, it can be difficult to learn such behavior. This is, in part, due to risk. For instance, throughout the learning process, an agent may have learned to avoid colliding with the ghosts. Without straying from this behavior, the agent will not learn that there are instances where it can safely chase the ghosts.In this paper, we consider an information-theoretic learning [6] approach for performing constrained stochastic searches that promote a continuum of risk-averse to risk-favoring agent behaviors during reinforcement learning. This, in turn, leads to a principled exploration of the state-action space that aids in the...

show abstract

Discovering Multimodal Behavior in Ms. Pac-Man Through Evolution of Modular Neural Networks

Cited by 36 publications

References 43 publications

Guiding Neuroevolution with Structural Objectives

Guiding Neuroevolution with Structural Objectives

Automatic Evolution of Multimodal Behavior with Multi-Brain HyperNEAT

Analysis of Agent Expertise in Ms. Pac-Man Using Value-of-Information-Based Policies

Contact Info

Product

Resources

About