MoleGuLAR: Molecule Generation using Reinforcement Learning with Alternating Rewards

Abstract

Design of new inhibitors for novel targets is a very important problem especially in the current scenario with the world being plagued by COVID-19. Conventional approaches undertaken to this end, like, high-throughput virtual screening require extensive combing through existing datasets in the hope of finding possible matches. In this study we propose a computational strategy for de novo generation of molecules with high binding affinities to the specified target. A deep generative model is built using a stack augmented recurrent neural network for initially generating drug like molecules and then it is optimized using reinforcement learning to start generating molecules with desirable properties--primarily the binding affinity. The reinforcement learning section of the pipeline is further extended to multi-objective optimization showcasing the model's ability to generate molecules with a wide variety of properties desirable for drug like molecules, like, LogP, Quantitative Estimate of Drug Likeliness etc.. For multi-objective optimization, we have devised a novel strategy in which the property being used to calculate the reward is changed periodically. In comparison to the conventional approach of taking a weighted sum of all rewards, this strategy shows enhanced ability to generate a significantly higher number of molecules with desirable properties.

Improving de novo molecular design with curriculum learning

“…3 Recently, deep learning has been applied towards more efficient methods of sampling chemical space such that it is possible to identify promising candidate molecules faster. Deep generative models using policy based reinforcement learning (RL) [4][5][6][7][8][9][10] , value based RL 11 , learning a molecular latent space 12 , and other methods including tree search 13 and genetic algorithms [14][15][16] have been proposed to generate molecules that possess a desired set of properties. In the policy based RL paradigm, an agent (a generative model) learns a policy (series of actions to take at given states) to generate molecules that maximize a reward which is typically computed based on a predefined reward function.…”

Improving De Novo Molecular Design with Curriculum Learning

“…In this section, we demonstrate that simple curricula, utilizing a single Curriculum Objective can accelerate agent productivity and generate compounds that satisfy a docking constraint, i.e., predicted to retain experimentally validated interactions (see Methods for experiment hyperparameters). 6,7,[13][14][15] Simulating a real-world application where one must allocate limited computational resources, baseline RL and CL performances are compared, given a maximum number of permitted production epochs (300), i.e., epochs that involve docking, as these are relatively computationally demanding. For CL, Curriculum Objectives are first applied to guide the agent and the number of permitted curriculum epochs is not limited, as these are computationally inexpensive (see Table S2).…”

“…3 Recently, deep learning has been applied towards more efficient methods of sampling chemical space such that it is possible to identify promising candidate molecules faster. Deep generative models using policy-based reinforcement learning (RL) [4][5][6][7][8][9][10] , value based RL 11 , learning a molecular latent space 12 , and other methods including tree search 13 and genetic algorithms [14][15][16] have been proposed to generate molecules that possess a desired set of properties. In the policy-based RL paradigm, an agent (a generative model) learns a policy (series of actions to take at given states) to generate molecules that maximize a reward which is typically computed based on a pre-defined reward function.…”

“…In the policy-based RL paradigm, an agent (a generative model) learns a policy (series of actions to take at given states) to generate molecules that maximize a reward which is typically computed based on a pre-defined reward function. [4][5][6][7][8][9][10] Often, physics-based approximations of binding affinity such as molecular docking are included as a component in the reward function in order to design candidate molecules with enhanced predicted potency. Given sufficiently long training time, these models can learn to generate molecules which satisfy the desired MPO objective.…”

Improving De Novo Molecular Design with Curriculum Learning

“…3 Recently, deep learning has been applied towards more efficient methods of sampling chemical space such that it is possible to identify promising candidate molecules faster. Deep generative models using policy based reinforcement learning (RL) [4][5][6][7][8][9][10] , value based RL 11 , learning a molecular latent space 12 , and other methods including tree search 13 and genetic algorithms [14][15][16] have been proposed to generate molecules that possess a desired set of properties. In the policy based RL paradigm, an agent (a generative model) learns a policy (series of actions to take at given states) to generate molecules that maximize a reward which is typically computed based on a predefined reward function.…”