Efficient Selectivity and Backup Operators in Monte-Carlo Tree Search

Abstract

Abstract. Monte-Carlo evaluation consists in estimating a position by averaging the outcome of several random continuations, and can serve as an evaluation function at the leaves of a min-max tree. This paper presents a new framework to combine tree search with Monte-Carlo evaluation, that does not separate between a min-max phase and a MonteCarlo phase. Instead of backing-up the min-max value close to the root, and the average value at some depth, a more general backup operator is defined that progressively changes from averaging to min-max as the number of simulations grows. This approach provides a fine-grained control of the tree growth, at the level of individual simulations, and allows efficient selectivity methods. This algorithm was implemented in a 9 × 9 Go-playing program, Crazy Stone, that won the 10th KGS computer-Go tournament.

Adding Expert Knowledge and Exploration in Monte-Carlo Tree Search

“…First, as well as for humans, all time scales of learning are important: offline knowledge (strategic rules and patterns) as in [7,5]; online information (i.e. analysis of a sequence by mental simulations) [10]; transient information (extrapolation as a guide for exploration).…”

“…Monte-Carlo Tree Search (MCTS [5,7,11]) is a recent tool for difficult planning tasks. Impressive results have already been produced in the case of the game of Go [7,10].…”

“…end while Return the move simulated most often from the root of T . The reader is referred to [5,7,11,10,9] for a detailed presentation of MCTS techniques and various scores. We will propose our current bandit formula in section 2.…”

Upper Confidence Trees with Short Term Partial Information

“…We assume in this section that the reader is familiar with the Monte-Carlo Tree Search (MCTS) and Upper Confidence Tree (UCT) literature [4,7,9]. We here focus on the experimental application of MCTS to acyclic GSA games.…”

“…2) has 11 millions of registered users. It's a Card Game, related to games like Pokemon or Magic, with Partial Observability, a small number of turns leading to fast games (often less than a minute) 4 . First, each player chooses a deck, which contains four cards (see Fig.…”