OGA-UCT: On-the-Go Abstractions in UCT

Abstract

Recent work has begun exploring the value of domain abstractions in Monte-Carlo Tree Search (MCTS) algorithms for probabilistic planning. These algorithms automatically aggregate symmetric search nodes (states or state-action pairs) saving valuable planning time. Existing algorithms alternate between two phases: (1) abstraction computation forcomputing node aggregations, and (2) modified MCTS that use aggregate nodes. We believe that these algorithms do not achieve the full potential of abstractions because of disjoint phases – e.g., it can take a while to recover from erroneous abstractions, or compute better abstractions based on newly found knowledge.In response, we propose On-the-Go Abstractions (OGA), a novel approach in which abstraction computation is tightlyintegrated into the MCTS algorithm. We implement these on top of UCT and name the resulting algorithm OGA-UCT.It has several desirable properties, including (1) rapid use of new information in modifying existing abstractions, (2) elimination of the expensive batch abstraction computationphase, and (3) focusing abstraction computation on important part of the sampled search space. We experimentally compare OGA-UCT against ASAP-UCT, a recent state-of-the-art MDP algorithm as well as vanilla UCT algorithm. We find that OGA-UCT is robust across a suite of planning competition and other MDP domains, and obtains up to 28 % quality improvements.

Mori-Zwanzig Approach for Belief Abstraction with Application to Belief Space Planning

“…However, the bisimulation metric is difficult to satisfy completely and is not robust to small changes in the transition probabilities and rewards Moerland et al (2023). Safe state abstraction methods avoid the non-Markovian problem by ignoring irrelevant state variables Andre and Russell (2002); Anand et al (2015), thus yield state abstraction while maintaining hierarchical optimality, i.e., optimality among all policies consistent with the partial program. One method to resolve this situation is to introduce an ad hoc weighting function (the aggregation probability), which functions like an occupancy probability for each concrete state given the abstract state.…”

Elastic Monte Carlo Tree Search

“…This group is gradually divided into different groups to refine the state abstraction. On-The-Go abstraction [13] updates the abstraction more frequently, avoiding the influence of delayed samples. A recent visit count is kept for each node and the abstraction mapping for a node is re-computed once the recent visit count reached a threshold.…”

“…The proposed method improved the agent's performance in the game Othello when combined with further action abstraction and tree pruning. [11] extended the application of approximate MDP homomorphism from state abstraction to state-action abstraction where the state-action pairs are grouped. In our paper, we adapt approximate MDP homomorphism from [9] to perform in the more challenging domain of strategy games, without providing game expert knowledge.…”

“…Existing methods [9], [11], [13] assume the generated state abstraction to be of high quality so they keep using the state abstraction until the search is finished. However, it is well known that approximate MDP homomorphism is a lossy abstraction, i.e., introduces imperfections in the search space [13].…”

Training Deep Reactive Policies for Probabilistic Planning Problems

“…Current state-of-theart planners for RDDL problems are based on online search, where at each step some combination of search and reasoning is used to select an action. For example, there are planners based on sample-based tree search (Keller and Eyerich 2012;Kolobov et al 2012;Bonet and Geffner 2012), symbolic variants (Cui et al 2015;Raghavan et al 2015;Anand et al 2016), and those that construct and solve integer linear programs at each step (Issakkimuthu et al 2015). These planners can require non-trivial computation time per step, which can make them inapplicable to problems that require fast decisions.…”