Enhancing Biomolecular Sampling with Reinforcement Learning: A Tree Search Molecular Dynamics Simulation Method

Abstract

This paper proposes a novel molecular simulation method, called tree search molecular dynamics (TS-MD), to accelerate the sampling of conformational transition pathways, which require considerable computation. In TS-MD, a tree search algorithm, called upper confidence bounds for trees, which is a type of reinforcement learning algorithm, is applied to sample the transition pathway. By learning from the results of the previous simulations, TS-MD efficiently searches conformational space and avoids being trapped in local stable structures. TS-MD exhibits better performance than parallel cascade selection molecular dynamics, which is one of the state-of-the-art methods, for the folding of miniproteins, Chignolin and Trp-cage, in explicit water.

Molecular simulations of amyloid beta assemblies

“…One of the main challenges of the following years of computational research in this field will be the possibility to combine the molecular dynamics technique with artificial intelligence methodologies. Several studies have already demonstrated the ability of machine learning methodologies to appropriate select collective variable for enhancing sampling of molecular simulations [79][80][81][82], Tree search molecular dynamics (TS-MD) [82], has also demonstrated to speed up the conformational sampling of protein transition pathways, which require considerable computational effort. By considering the results of the previous simulations, reinforcement learning efficiently sample new conformational states and avoids being trapped in local-free energy minima.…”

“…The main limitation of these approaches is mostly related with insufficient sampling, which often limits the ability of computer simulations to fully handle proteins' rough energy landscapes, with many local minima separated by high-energy barriers. Within this framework, one of the main challenges of the following years of computational research in this field will be the possibility to combine the Molecular Dynamics technique with artificial intelligence methodologies [79][80][81][82], in order to sample relevant millisecond to second timescales. This research advance will provide the possibility to sample the complex transition from oligomers to fibrils and vice-versa.…”

Structure-based in silico approaches for drug discovery against Mycobacterium tuberculosis

“…Several methods have been applied to enhance the sampling of protein conformations during MD simulations, either through biasing of ongoing MD simulations [120] or through selection of starting-point protein conformations for short MD simulations [121] , [122] . Anncolvar uses machine learning to approximate CVs for metadynamic MD simulations, which would otherwise be too computationally expensive to apply during a simulation, such as molecular surface area calculations [120] .…”

Probing conformational dynamics of antibodies with geometric simulations

“…Since the conformations of interest often lack either starting or final crystal structures, FRODAN's targeting in this case is less applicable. To overcome this difficulty, FRO-DAN non-targeted mode can be combined with search algorithms, for example Monte Carlo Tree Search (MCTS) -a type of reinforcement learning algorithm that has demonstrated impressive performance at various classes of tasks including protein folding [17,18]. The strength of MCTS is the ability to balance between exploration and exploitation when search is expanded.…”