Temporal-difference search in computer Go

Abstract

Temporal-difference learning is one of the most successful and broadly applied solutions to the reinforcement learning problem; it has been used to achieve master-level play in chess, checkers and backgammon. The key idea is to update a value function from episodes of real experience, by bootstrapping from future value estimates, and using value function approximation to generalise between related states. Monte-Carlo tree search is a recent algorithm for high-performance search, which has been used to achieve masterlevel play in Go. The key idea is to use the mean outcome of simulated episodes of experience to evaluate each state in a search tree. We introduce a new approach to highperformance search in Markov decision processes and two-player games. Our method, temporal-difference search, combines temporal-difference learning with simulation-based search. Like Monte-Carlo tree search, the value function is updated from simulated experience; but like temporal-difference learning, it uses value function approximation and bootstrapping to efficiently generalise between related states. We apply temporal-difference search to the game of 9 × 9 Go, using a million binary features matching simple patterns of stones. Without any explicit search tree, our approach outperformed an unenhanced MonteCarlo tree search with the same number of simulations. When combined with a simple alpha-beta search, our program also outperformed all traditional (pre-Monte-Carlo) search and machine learning programs on the 9 × 9 Computer Go Server.

Q-CP: Learning Action Values for Cooperative Planning

“…Thus improving the simulation time and computational load. To this end, we compare our solution against TD-search [2] and both a vanilla-UCT and random-UCT implementations. We refer to vanilla-UCT as the standard UCT algorithm that, at each iteration, expands every possible action in A j , for every agent j.…”

“…Nevertheless, they show difficulties in relating similar states (i.e. nodes of the search tree) [2]. Here, we focus on the problem of cooperative general-sum stochastic games [3], where each agent runs its own learning process.…”

Game playing

“…Some examples include: One of the original proposals for Monte Carlo search was to improve the strength of TD‐Gammon, using its learned neural network evaluation function to bias move choice during playouts . Furthermore, some of the research on Monte Carlo tree search for Go originates in the reinforcement learning community that developed TD learning; it is possible to view Monte Carlo tree search as a kind of reinforcement learning that takes place in real time while analyzing a position to choose a move Monte Carlo tree search has been established as the leading method in other games including Hex and Lines of Action, and has been used to close in on human expert levels of play in Havannah .…”

Reinforcement learning analysis for a minimum time balance problem

“…Balakrishna et al (2010) used the value function learning accurately to predict taxi-out times in airports. Similarly, Silver et al (2012) introduced a new approach in the game Go by implementing both value function approximation and bootstrapping to obtain high performance policy search. They showed that this approach outperforms to Monte Carlo Tree search.…”