Consistency Modifications for Automatically Tuned Monte-Carlo Tree Search

Abstract

Abstract. Monte-Carlo Tree Search algorithms (MCTS [4,6]), including upper confidence trees (UCT [9]), are known for their impressive ability in high dimensional control problems. Whilst the main testbed is the game of Go, there are increasingly many applications [13,12,7]; these algorithms are now widely accepted as strong candidates for highdimensional control applications. Unfortunately, it is known that for optimal performance on a given problem, MCTS requires some tuning; this tuning is often handcrafted or automated, with in some cases a loss of consistency, i.e. a bad behavior asymptotically in the computational power. This highly undesirable property led to a stupid behavior of our main MCTS program MoGo in a real-world situation described in section 3. This is a big trouble for our several works on automatic parameter tuning [3] and the genetic programming of new features in MoGo. We will see in this paper:-A theoretical analysis of MCTS consistency; -Detailed examples of consistent and inconsistent known algorithms; -How to modify a MCTS implementation in order to ensure consistency, independently of the modifications to the "scoring" module (the module which is automatically tuned and genetically programmed in MoGo); -As a by product of this work, we'll see the interesting property that some heavily tuned MCTS implementations are better than UCT in the sense that they do not visit the complete tree (whereas UCT asymptotically does), whilst preserving the consistency at least if "consistency" modifications above have been made.

A Survey of Monte Carlo Tree Search Methods

“…Silver describes the technique of simulation balancing using gradient descent to bias the policy during simulations [203]. While it has been observed that improving the simulation policy does not necessarily lead to strong play [92], Silver and Tesauro demonstrate techniques for learning a simulation policy that works well with MCTS to produce balanced 19 if not strong play [203].…”

“…In terms of board games such as Go, a pattern is a small non-empty section of the board or a logical test upon it. Patterns may also encode additional information such as the player to move, and are typically incorporated into 19. Games in which errors by one player are on average cancelled out by errors by the opponent on their next move [203].…”

Monte Carlo Search Algorithm Discovery for Single-Player Games

“…For instance, Perrick et al [27] employ a grid search approach combined with self-playing, Chaslot et al [28] use cross entropy as a search method to tune an agent playing Go, Coulom [29] presents a generic black-box optimization method based on local quadratic regression, Maes et al [30] use estimation distribution algorithms with Gaussian distributions, Chapelle and Li [31] use Thompson sampling, and Bourki et al [32] uses, as in the present paper, a multi-armed bandit approach. The paper [33] studies the influence of the tuning of MCS algorithms on their asymptotic consistency and shows that pathological behavior may occur with tuning. It also proposes a tuning method to avoid such behavior.…”

A Principled Method for Exploiting Opening Books

“…Finally, it has been suggested in [8,3] to use a regularized form of the winning rate; we propose this in the "regularized" algorithm 1 (Alg. 5).…”