Reinforcement learning of rare diffusive dynamics

Abstract

We present a method to probe rare molecular dynamics trajectories directly using reinforcement learning. We consider trajectories that are conditioned to transition between regions of configuration space in finite time, like those relevant in the study of reactive events, as well as trajectories exhibiting rare fluctuations of time-integrated quantities in the long time limit, like those relevant in the calculation of large deviation functions. In both cases, reinforcement learning techniques are used to optimize an added force that minimizes the Kullback-Leibler divergence between the conditioned trajectory ensemble and a driven one. Under the optimized added force, the system evolves the rare fluctuation as a typical one, affording a variational estimate of its likelihood in the original trajectory ensemble. Low variance gradients employing value functions are proposed to increase the convergence of the optimal force. The method we develop employing these gradients leads to efficient and accurate estimates of both the optimal force and the likelihood of the rare event for a variety of model systems.

Generating stochastic trajectories with global dynamical constraints

“…There has been recent progress in developing efficient numerical algorithms, using importance sampling, that allow to measure such rare events with probabilities as small as 10 −100 [83,84]! In the context of systems out-of-equilibrium, new algorithms, inspired by reinforcement learning and machine learning approaches, have been developed [19][20][21]. Here, we have provided a simple single-particle stochastic process in the presence of time-integrated dynamical constraint and we have shown that it can be generated very simply using an effective but exact Langevin equation.…”

“…A naive solution would be to generate free paths and discard the ones that do not satisfy the constraint. Unfortunately, this method turns out to be computationally wasteful as the trajectories satisfying the constraint are typically rare [15][16][17][18][19][20][21][22] and therefore difficult to obtain. Fortunately, for the case of Brownian motion, there exist several efficient methods, based on the so-called Doob transform [23,24], to generate particular types of constrained trajectories.…”

Optimal Resetting Brownian Bridges

“…Our first goal is to construct a rejection-free algorithm to generate an RBB with the correct statistical weight. Generating constrained stochastic Markov processes was initially studied in the probability literature [73,74] and more recently it has emerged as a vibrant research area by itself in the context of sampling rare/constrained trajectories with applications ranging from chemistry and biology all the way to particle physics [75][76][77][78][79][80][81][82][83][84][85][86][87][88][89][90][91][92][93][94]. In the context of the RBB, a naive solution would be to generate free RBM paths and discard the ones that do not satisfy the bridge constraint x B (t f ) = 0.…”

Direct evaluation of rare events in active matter from variational path sampling

“…The optimization problem maps onto the computation of a cumulant generating function for the statistics of the indicator h B (t f ) studied previously, 26,34 with the short trajectories starting from a steady-state distribution in the initial state. As such we can employ generalizations of recent reinforcement learning procedures to efficiently estimate the gradients of the loss function with respect to the variational parameters.…”

“…As such we can employ generalizations of recent reinforcement learning procedures to efficiently estimate the gradients of the loss function with respect to the variational parameters. Specifically, we modify the Monte-Carlo Value Baseline (MCVB) algorithm 34 which performs a stochastic gradient descent to optimize c (i) pq . We add two preconditioning steps over the MCVB algorithm.…”