Multiagent Reinforcement Learning-Based Adaptive Sampling for Conformational Dynamics of Proteins

Abstract

Machine learning is increasingly applied to improve the efficiency and accuracy of molecular dynamics (MD) simulations. Although the growth of distributed computer clusters has allowed researchers to obtain higher amounts of data, unbiased MD simulations have difficulty sampling rare states, even under massively parallel adaptive sampling schemes. To address this issue, several algorithms inspired by reinforcement learning (RL) have arisen to promote exploration of the slow collective variables (CVs) of complex systems. Nonetheless, most of these algorithms are not well-suited to leverage the information gained by simultaneously sampling a system from different initial states (e.g., a protein in different conformations associated with distinct functional states). To fill this gap, we propose two algorithms inspired by multiagent RL that extend the functionality of closely related techniques (REAP and TSLC) to situations where the sampling can be accelerated by learning from different regions of the energy landscape through coordinated agents. Essentially, the algorithms work by remembering which agent discovered each conformation and sharing this information with others at the action-space discretization step. A stakes function is introduced to modulate how different agents sense rewards from discovered states of the system. The consequences are three-fold: (i) agents learn to prioritize CVs using only relevant data, (ii) redundant exploration is reduced, and (iii) agents that obtain higher stakes are assigned more actions. We compare our algorithm with other adaptive sampling techniques (least counts, REAP, TSLC, and AdaptiveBandit) to show and rationalize the gain in performance.

Conformational Stability and Denaturation Processes of Proteins Investigated by Electrophoresis under Extreme Conditions

“…One of the possible ways to relieve computational costs is to transition from all-atom to coarse-grained models of proteins [ 102 , 114 , 126 ]. Other options are the use of accelerated molecular dynamics methods [ 84 , 127 ] and enhanced sampling techniques [ 84 , 128 , 129 , 130 , 131 ], including machine learning approaches [ 132 , 133 , 134 ].…”

Distinct Activation Mechanisms Regulate Subtype Selectivity of Cannabinoid Receptors

“…67 Thus, to capture the entire protein conformational ensemble, adaptive sampling protocol was implemented. [67][68][69][70] Adaptive sampling is a well established sampling technique used for studying ligand binding, [71][72][73] protein conformational change 63,[74][75][76] and ligand selectivity. 77 S7 and S8).…”

Thirty years of molecular dynamics simulations on posttranslational modifications of proteins

“…Methodology has been proposed for making seed state selection as statistically unbiased as possible, as adaptive sampling can use any frame from a previous trajectory as a starting seed for the next round. [240][241][242][243][244] Because adaptive sampling regimes incorporate selection of multiple trajectories per round based on preexisting landscape coverage, they can improve statistical characterization of rare high energy regions between minima; traditionally, long trajectories struggle to capture rare events in more than one transition. Adaptive sampling workflowsand their representation through FELs -are easily amenable to Markov state model (MSM) generation.…”