Learning to Play Using Low-Complexity Rule-Based Policies: Illustrations through Ms. Pac-Man

Abstract

In this article we propose a method that can deal with certain combinatorial reinforcement learning tasks. We demonstrate the approach in the popular Ms. Pac-Man game. We define a set of high-level observation and action modules, from which rule-based policies are constructed automatically. In these policies, actions are temporally extended, and may work concurrently. The policy of the agent is encoded by a compact decision list. The components of the list are selected from a large pool of rules, which can be either hand-crafted or generated automatically. A suitable selection of rules is learnt by the cross-entropy method, a recent global optimization algorithm that fits our framework smoothly. Cross-entropy-optimized policies perform better than our hand-crafted policy, and reach the score of average human players. We argue that learning is successful mainly because (i) policies may apply concurrent actions and thus the policy space is sufficiently rich, (ii) the search is biased towards low-complexity policies and therefore, solutions with a compact description can be found quickly if they exist.

Using the online cross-entropy method to learn relational policies for playing different games

“…As a reward comparison, the agent presented in [8] achieves 8186 points on average using CE optimisation of hand-coded rules and 6382 points using CE optimisation of random rules. However, note that CERRLA was designed to learn in any relational environment (resulting in a loss in performance), whereas the agent in [8] was designed specifically for playing Ms. Pac-Man.…”

“…However, note that CERRLA was designed to learn in any relational environment (resulting in a loss in performance), whereas the agent in [8] was designed specifically for playing Ms. Pac-Man. Furthermore, the Ms. Pac-Man environments used are likely to be different in execution.…”

“…The crossentropy method has previously been successfully applied to both Tetris [9], [10] and Ms. Pac-Man [8], though in each case the learning algorithm was optimised to match the environment.…”

Optimized look‐ahead tree policies: a bridge between look‐ahead tree policies and direct policy search

“…Another example for a policy parametrization is to use domain-specific building blocks, such as motor primitives, as it was done in [27] and [29] to optimize the gait of the AIBO quadrupedal robot. A different kind of policy representation is used in [50] to learn a policy for the game of Ms. Pac-Man; here the policy is represented by a list of domain-specific parameterized rules. Section 4 described a third option for the global optimization part: Gaussian process optimization [35,3].…”

An influence map model for playing Ms. Pac-Man

“…Recently, Szita and Lorincz [10] proposed a different approach to playing Ms. Pac-Man. The aim is to develop a simple rule-based policy, where rules are organized into action modules and a decision about which direction to move is made based on priorities assigned to the modules in the agent.…”