Online Planning for Large Markov Decision Processes with Hierarchical Decomposition

Abstract

Markov decision processes (MDPs) provide a rich framework for planning under uncertainty. However, exactly solving a large MDP is usually intractable due to the "curse of dimensionality"-the state space grows exponentially with the number of state variables. Online algorithms tackle this problem by avoiding computing a policy for the entire state space. On the other hand, since online algorithm has to find a near-optimal action online in almost real time, the computation time is often very limited. In the context of reinforcement learning, MAXQ is a value function decomposition method that exploits the underlying structure of the original MDP and decomposes it into a combination of smaller subproblems arranged over a task hierarchy. In this article, we present MAXQ-OP-a novel online planning algorithm for large MDPs that utilizes MAXQ hierarchical decomposition in online settings. Compared to traditional online planning algorithms, MAXQ-OP is able to reach much more deeper states in the search tree with relatively less computation time by exploiting MAXQ hierarchical decomposition online. We empirically evaluate our algorithm in the standard Taxi domain-a common benchmark for MDPs-to show the effectiveness of our approach. We have also conducted a long-term case study in a highly complex simulated soccer domain and developed a team named WrightEagle that has won five world champions and five runners-up in the recent 10 years of RoboCup Soccer Simulation 2D annual competitions. The results in the RoboCup domain confirm the scalability of MAXQ-OP to very large domains. ACM Reference Format:Aijun Bai, Feng Wu, and Xiaoping Chen. 2015. Online planning for large Markov decision processes with hierarchical decomposition.

Fractals2019: Combinatorial Optimisation with Dynamic Constraint Annealing

“…It is evident that the local maximum attained by test 34 does not satisfy all these local constraints, highlighting the need for a second, constraint satisfaction, phase. The algorithm continues with the best solution X 34 = {2, 3,5,4,8,10,11,9,6, 7}, reevaluated after 16000 games in order to ensure a higher precision, attaining f 34 = −3.14496, and then generates the candidates closer to and within the constrained sub-space, as shown in Table 4, for a specified number of tests, e.g., 10 additional tests. The maximum attained at the end of this phase is given by X 44 = {5, 4, 2, 3, 7, 6, 8, 10, 11, 9} with f 44 = −2.95471, reducing the goal deficit by further 0.2.…”

“…The RoboCup Soccer 2D Simulation League provides a rich dynamic environment, facilitated by the RoboCup Soccer Simulator (RCSS), aimed to test advances in decentralised collective behaviours of autonomous agents. The challenges include concurrent adversarial actions, computational nondeterminism, noise and latency in asynchronous perception and actuation, and limited processing time [3,5,7,29,37,38,42,43,46]. The League progress has been supported by several important base code releases, covering both low-level skills and standardised world models of simulated agents [1,22,45,47].…”

The Decision-Making Framework of WrightEagle, the RoboCup 2013 Soccer Simulation 2D League Champion Team

“…The WrightEagle team is developed base on the Markov Decision Processes (MDPs) framework [1] with the MAXQ hierarchical structure [2] and heuristic approximate online planning techniques in the past years [3] [4].…”

“…For instance, at Robocup 2013, there were a total of 20 teams from 9 countries entering the 2D simulation competition 1 . Taking into account the scale of the 2D simulation competition, making good decisions against so many different opponents is a very challenging task and dealing with complex and varied situations is very difficult [3]. This paper introduces the RoboCup 2013 Soccer Simulation 2D League champion team, WrightEagle, that has won 4 champions and 5 runners-up in the past 9 years.…”

Heuristic Function Negotiation for Markov Decision Process and Its Application in UAV Simulation

“…Completely ignoring domain information can cause the learning algorithms inefficiency. Dietterich presented the MAXQ hierarchical RL method, which is based on a semi-Markov decision process (SMDP) [5]- [7]. MAXQ employs user knowledge by a layered approach, and it divides the overall learning task into a number of reusable sub-Markov decision processes to speed up the learning.…”