An Analysis of Virtual Loss in Parallel MCTS

Mirsoleimani, S. Ali; Plaat, Aske; Herik, Jaap van den; Vermaseren, J.A.M.

doi:10.5220/0006205806480652

Cited by 6 publications

(7 citation statements)

References 4 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In our future research we will look into additional techniques, promising a speed-up which scales better with the number of threads [20], as the standard techniques for parallelization do not address the trade of between exploration and exploitation well enough [19], [20], [25].…”

Section: Discussionmentioning

confidence: 99%

“…In addition, measures such as virtual loss [6] improve the effectiveness of tree parallelization, by increasing the likelihood that multiple threads traverse different paths. While virtual loss has been an adopted concept [2], more recent studies suggest that the increase in effectiveness is merely a trade-off with time efficiency [19].…”

Section: B Parallelizationmentioning

confidence: 99%

See 1 more Smart Citation

Parallelization of Monte Carlo Tree Search in Continuous Domains

Kurzer¹,

Christoph²,

Zöllner³

2020

Preprint

View full text Add to dashboard Cite

Monte Carlo Tree Search (MCTS) has proven to be capable of solving challenging tasks in domains such as Go, chess and Atari. Previous research has developed parallel versions of MCTS, exploiting today's multiprocessing architectures. These studies focused on versions of MCTS for the discrete case. Our work builds upon existing parallelization strategies and extends them to continuous domains. In particular, leaf parallelization and root parallelization are studied and two final selection strategies that are required to handle continuous states in root parallelization are proposed. The evaluation of the resulting parallelized continuous MCTS is conducted using a challenging cooperative multi-agent system trajectory planning task in the domain of automated vehicles.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: B Parallelizationmentioning

confidence: 99%

Parallelization of Monte Carlo Tree Search in Continuous Domains

Kurzer¹,

Christoph²,

Zöllner³

2020

Preprint

View full text Add to dashboard Cite

show abstract

“…This could be observed from our experiments in Section 5. In contrast, although the virtual loss used in TreeP could encourage exploration diversity, this hard additive penalty could cause exploitatin failure: workers will be less likely to co-simulating the same node even when they are certain that it is optimal (Mirsoleimani et al, 2017). RootP tries to avoid these issues by letting workers perform an independent tree search.…”

Section: The Benefits Of Watching Unobserved Samplesmentioning

confidence: 99%

“…Additionally, architecture side improvements such as using pipeline (Mirsoleimani et al, 2018b) or lock-free structure (Mirsoleimani et al, 2018a) speedup the algorithm significantly. However, though being able to increase diversity, virtual loss degrades the performance under even four workers (Mirsoleimani et al, 2017;Bourki et al, 2010).…”

Section: Related Workmentioning

confidence: 99%

Watch the Unobserved: A Simple Approach to Parallelizing Monte Carlo Tree Search

Liu,

Chen,

et al. 2018

Preprint

View full text Add to dashboard Cite

Monte Carlo Tree Search (MCTS) algorithms have achieved great success on many challenging benchmarks (e.g., Computer Go). However, they generally require a large number of rollouts, making their applications to planning costly. Furthermore, it is also extremely challenging to parallelize MCTS due to its inherent sequential nature: each rollout heavily relies on the statistics (e.g., node visitation counts) estimated from previous simulations to achieve an effective explorationexploitation tradeoff. In spite of these difficulties, we develop an algorithm, P-UCT 1 , to effectively parallelize MCTS, which achieves linear speedup and exhibits negligible performance loss with an increasing number of workers. The key idea in P-UCT is a set of statistics that we introduce to track the number of ongoing yet incomplete simulation queries (named as unobserved samples). These statistics are used to modify the UCT tree policy in the selection steps in a principled manner to retain effective exploration-exploitation tradeoff when we parallelize the most time-consuming expansion and simulation steps. Experimental results on a proprietary benchmark and the public Atari Game benchmark demonstrate the near-optimal linear speedup and the superior performance of P-UCT when compared to these existing techniques.

show abstract

“…Despite their extensive usage, the performance of parallel MCTS algorithms [12] is not systematically understood from a theoretical perspective. There are empirical studies on the advantages (e.g., [12,13,14]) and disadvantages (e.g., [15,16,17]) of existing approaches. However, they are mainly algorithm-specific analysis, which provides less systematic design principles on effective MCTS parallelization.…”

Section: Introductionmentioning

confidence: 99%

On Effective Parallelization of Monte Carlo Tree Search

Liu¹,

Liang²,

Ji³

et al. 2020

Preprint

View full text Add to dashboard Cite

Despite its groundbreaking success in Go and computer games, Monte Carlo Tree Search (MCTS) is computationally expensive as it requires a substantial number of rollouts to construct the search tree, which calls for effective parallelization. However, how to design effective parallel MCTS algorithms has not been systematically studied and remains poorly understood. In this paper, we seek to lay its first theoretical foundations, by examining the potential performance loss caused by parallelization when achieving a desired speedup. In particular, we focus on studying the conditions under which the performance loss (measured in excess regret) vanishes over time. To this end, we propose a general parallel MCTS framework that can be specialized to major existing parallel MCTS algorithms. We derive two necessary conditions for the algorithms covered by the general framework to have vanishing excess regret (i.e. excess regret converges to zero as the total number of rollouts grows). We demonstrate the effectiveness of the necessary conditions by showing that, for depth-2 search trees, the recently developed WU-UCT algorithm satisfies both necessary conditions and has provable vanishing excess regret. Finally, we perform empirical studies to closely examine the necessary conditions under the general tree search setting (with arbitrary tree depth). It shows that the topological discrepancy between the search trees constructed by the parallel and the sequential MCTS algorithms is the main reason for the performance loss.

show abstract

An Analysis of Virtual Loss in Parallel MCTS

Cited by 6 publications

References 4 publications

Parallelization of Monte Carlo Tree Search in Continuous Domains

Parallelization of Monte Carlo Tree Search in Continuous Domains

Watch the Unobserved: A Simple Approach to Parallelizing Monte Carlo Tree Search

On Effective Parallelization of Monte Carlo Tree Search

Contact Info

Product

Resources

About