A model-based cell decomposition approach to on-line pursuit-evasion path planning and the video game Ms. Pac-Man

Foderaro, Greg; Swingler, Ashleigh; Ferrari, Silvia

doi:10.1109/cig.2012.6374167

Cited by 12 publications

(14 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Robles and Lucas [33] adapted traditional game-tree search to work in Ms. Pac-Man, and in the most recent competition, Ikehata and Ito [34] used Monte-Carlo Tree Search (MCTS) in their winning entry. The competition has not been run since 2011, but in 2012 Foderaro et al [35] painstakingly modeled the idiosyncratic details of the ghosts’ behaviors 2 and decomposed the corridors and junctions of the mazes into cells in order to learn a decision-tree-based policy that outperformed MCTS (though this success is likely due to their detailed, human-supplied ghost model).…”

Section: Related Workmentioning

confidence: 99%

Discovering Multimodal Behavior in Ms. Pac-Man Through Evolution of Modular Neural Networks

Schrum

Miikkulainen

2016

IEEE Trans. Comput. Intell. AI Games

View full text Add to dashboard Cite

Ms. Pac-Man is a challenging video game in which multiple modes of behavior are required: Ms. Pac-Man must escape ghosts when they are threats and catch them when they are edible, in addition to eating all pills in each level. Past approaches to learning behavior in Ms. Pac-Man have treated the game as a single task to be learned using monolithic policy representations. In contrast, this paper uses a framework called Modular Multi-objective NEAT (MM-NEAT) to evolve modular neural networks. Each module defines a separate behavior. The modules are used at different times according to a policy that can be human-designed (i.e. Multitask) or discovered automatically by evolution. The appropriate number of modules can be fixed or discovered using a genetic operator called Module Mutation. Several versions of Module Mutation are evaluated in this paper. Both fixed modular networks and Module Mutation networks outperform monolithic networks and Multitask networks. Interestingly, the best networks dedicate modules to critical behaviors (such as escaping when surrounded after luring ghosts near a power pill) that do not follow the customary division of the game into chasing edible and escaping threat ghosts. The results demonstrate that MM-NEAT can discover interesting and effective behavior for agents in challenging games.

show abstract

Section: Related Workmentioning

confidence: 99%

Discovering Multimodal Behavior in Ms. Pac-Man Through Evolution of Modular Neural Networks

Schrum

Miikkulainen

2016

IEEE Trans. Comput. Intell. AI Games

View full text Add to dashboard Cite

show abstract

“…The more heuristic-based approaches like the ones in [18][19][20], in contrast, do not necessarily possess these guarantees. The same is largely true for Monte Carlo tree search [27][28][29][31][32][33]. Convergence for many of these tree-search procedures is, currently, only guaranteed for simplified problem domains and for certain exploration strategies [30].…”

Section: Introductionmentioning

confidence: 99%

“…More recently, Samothrakis et al [29] and Pepels et al [31] applied Monte Carlo tree search to create high-performing agents. Foderaro et al [32,33] relied on tree searches as well. They first decompose the environment into a series of convex cells, which outlined locations where the agent could travel.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Analysis of Agent Expertise in Ms. Pac-Man Using Value-of-Information-Based Policies

Sledge

Prı́ncipe

2019

IEEE Trans. Games

View full text Add to dashboard Cite

Conventional reinforcement learning methods for Markov decision processes rely on weakly-guided, stochastic searches to drive the learning process. It can therefore be difficult to predict what agent behaviors might emerge. In this paper, we consider an information-theoretic cost function for performing constrained stochastic searches that promote the formation of risk-averse to risk-favoring behaviors. This cost function is the value of information, which provides the optimal trade-off between the expected return of a policy and the policy's complexity; policy complexity is measured by number of bits and controlled by a single hyperparameter on the cost function. As the policy complexity is reduced, the agents will increasingly eschew risky actions. This reduces the potential for high accrued rewards. As the policy complexity increases, the agents will take actions, regardless of the risk, that can raise the long-term rewards. The obtainable reward depends on a single, tunable hyperparameter that regulates the degree of policy complexity.We evaluate the performance of value-of-information-based policies on a stochastic version of Ms. Pac-Man. A major component of this paper is the demonstration that ranges of policy complexity values yield different game-play styles and explaining why this occurs. We also show that our reinforcementlearning search mechanism is more efficient than the others we utilize. This result implies that the value of information theory is appropriate for framing the exploitation-exploration trade-off in reinforcement learning.Index Terms-Value of information, constrained search, reinforcement learning, information theory Isaac J. Sledge is with the 2 objective of the agent is to clear the environment of pellets while navigating around the ghosts. However, after activating certain power-ups, the ghosts become vulnerable for a brief period of time. The agent can consume these ghosts for a score boost.The switch in ghost dynamics necessitates a change in the game-play strategy, since multiple distinct modes of behavior are required under different conditions [4,5]. Despite the need for multi-modal behaviors, conventional reinforcement-learning approaches have focused on constructing monolithic policies. Such policies would implement the same agent behaviors regardless of the vulnerability of the ghosts. Although it is possible to represent multimodal behavior with these policies, it can be difficult to learn such behavior. This is, in part, due to risk. For instance, throughout the learning process, an agent may have learned to avoid colliding with the ghosts. Without straying from this behavior, the agent will not learn that there are instances where it can safely chase the ghosts.In this paper, we consider an information-theoretic learning [6] approach for performing constrained stochastic searches that promote a continuum of risk-averse to risk-favoring agent behaviors during reinforcement learning. This, in turn, leads to a principled exploration of the state-action space that aids in the...

show abstract

“…Many approaches have been evaluated in this domain: influence maps [28], gametree search [17], decision trees [9], and Monte-Carlo Tree Search (MCTS) [10]. A common conclusion is that the quality of any method is greatly affected by the quality of the screen-capture procedure used to assess the game state.…”

Section: Ms Pac-man Researchmentioning

confidence: 99%

Evolving multimodal behavior with modular neural networks in Ms. Pac-Man

Schrum

Miikkulainen

2014

Proceedings of the 2014 Annual Conference on Genetic and Evolutionary Computation

View full text Add to dashboard Cite

Ms. Pac-Man is a challenging video game in which multiple modes of behavior are required to succeed: Ms. Pac-Man must escape ghosts when they are threats, and catch them when they are edible, in addition to eating all pills in each level. Past approaches to learning behavior in Ms. Pac-Man have treated the game as a single task to be learned using monolithic policy representations. In contrast, this paper uses a framework called Modular Multiobjective NEAT to evolve modular neural networks. Each module defines a separate policy; evolution discovers these policies and when to use them. The number of modules can be fixed or learned using a new version of a genetic operator, called Module Mutation, which duplicates an existing module that can then evolve to take on a distinct behavioral identity. Both the fixed modular networks and Module Mutation networks outperform traditional monolithic networks. More interestingly, the best modular networks dedicate modules to critical behaviors that do not follow the customary division of the game into chasing edible and escaping threatening ghosts.

show abstract

A model-based cell decomposition approach to on-line pursuit-evasion path planning and the video game Ms. Pac-Man

Cited by 12 publications

References 12 publications

Discovering Multimodal Behavior in Ms. Pac-Man Through Evolution of Modular Neural Networks

Discovering Multimodal Behavior in Ms. Pac-Man Through Evolution of Modular Neural Networks

Analysis of Agent Expertise in Ms. Pac-Man Using Value-of-Information-Based Policies

Evolving multimodal behavior with modular neural networks in Ms. Pac-Man

Contact Info

Product

Resources

About