Generalized Mean Estimation in Monte-Carlo Tree Search

Abstract

<jats:p>We consider Monte-Carlo Tree Search (MCTS) applied to Markov Decision Processes (MDPs) and Partially Observable MDPs (POMDPs), and the well-known Upper Confidence bound for Trees (UCT) algorithm. In UCT, a tree with nodes (states) and edges (actions) is incrementally built by the expansion of nodes, and the values of nodes are updated through a backup strategy based on the average value of child nodes. However, it has been shown that with enough samples the maximum operator yields more accurate node value estimates than averaging. Instead of settling for one of these value estimates, we go a step further proposing a novel backup strategy which uses the power mean operator, which computes a value between the average and maximum value. We call our new approach Power-UCT, and argue how the use of the power mean operator helps to speed up the learning in MCTS. We theoretically analyze our method providing guarantees of convergence to the optimum. Finally, we empirically demonstrate the effectiveness of our method in well-known MDP and POMDP benchmarks, showing significant improvement in performance and convergence speed w.r.t. state of the art algorithms.</jats:p>

A Unified Perspective on Value Backup and Exploration in Monte-Carlo Tree Search

“…Finally, we provide a theory of the use of α-divergence in MCTS for backup and exploration. Remarkably, we show that our theoretical framework unifies our two proposed methods Power-UCT (Dam et al, 2019) and entropy regularization (Dam et al, 2021), that can be obtained for particular choices of the value of α. In the general case where α is considered a real number greater than 0, we show that tuning α directly influences the navigation and backup phases of the tree search, providing a unique powerful mathematical formulation to effectively balance between exploration and exploitation in MCTS.…”

Monte-Carlo Robot Path Planning

“…Notably, we propose MCPP as a general MCTS-based framework for robotic path planning. MCPP can incorporate different exploration strategies [25], [26] to continuous actions, adapting, subsequently, the convergence rates for MCPP.…”

“…Power-UCT [25], an improvement over UCT, solves the problem of the underestimation of the average mean and the max-backup operators in MCTS by proposing the use of power mean as the backup operator. Power-UCT has a polynomial convergence rate for choosing the optimal action at the root node.…”

“…Power-UCT has a polynomial convergence rate for choosing the optimal action at the root node. TENTS [26] is derived as a result of Tsallis entropy regularization in MCTS. TENTS has an exponential convergence rate at the root node, which is faster than Power-UCT and UCT.…”

“…We continue by proposing different exploration strategies in MCPP for robotic path planning. In particular, we build on top of our prior work on power-mean UCT (Power-UCT) [25] and convex regularization with Tsallis Entropy Monte-Carlo Planning (TENTS) [26], integrating them in MCPP. We provide various experimental evaluations of MCPP, initially in MDP environments for completeness and thereafter in challenging POMDP tasks in 2D and 3D while planning with a 7-DOF robot arm.…”