Augmented Hill-Climb increases reinforcement learning efficiency for language-based de novo molecule generation

Abstract

A plethora of AI-based techniques now exists to conduct de novo molecule generation that can devise molecules conditioned towards a particular endpoint in the context of drug design. One popular approach is using reinforcement learning to update a recurrent neural network or language-based de novo molecule generator. However, reinforcement learning can be inefficient, sometimes requiring up to 10^5 molecules to be sampled to optimize more complex objectives, which poses a limitation when using computationally expensive scoring functions like docking or computer-aided synthesis planning models. In this work, we propose a reinforcement learning strategy called Augmented Hill-Climb based on a simple, hypothesis-driven hybrid between REINVENT and Hill-Climb that improves sample-efficiency by addressing the limitations of both currently used strategies. We compare its ability to optimize several docking tasks with REINVENT and benchmark this strategy against other commonly used reinforcement learning strategies including REINFORCE, REINVENT (version 1 & 2), Hill-Climb and best agent reminder. We find that optimization ability is improved ~1.5-fold and sample-efficiency is improved ~45-fold compared to REINVENT while still delivering appealing chemistry as output. Diversity filters were used, and their parameters were tuned to overcome observed failure modes that take advantage of certain diversity filter configurations. Lastly, we find that Augmented Hill-Climb outperforms the other reinforcement learning strategies used on six tasks, especially in the early stages of training or for more difficult objectives. Overall, we hence show that AHC improves sample-efficiency for language-based de novo molecule generation conditioning via reinforcement learning, compared to the current state-of-the-art. This makes more computationally expensive scoring functions, such as docking, more accessible on a relevant timescale.

Augmented Memory: Sample-Efficient Generative Molecular Design with Reinforcement Learning

“…The metric used is the area under the curve (AUC) for the top 10 molecules. We note that Thomas et al 43 proposed a modified AUC Top-10 metric that incorporates diversity, but we omit comparison as the formulation can be subjective. The current Top AUC-10 metric assesses sample efficiency, which is our focus.…”

“…We expand the PMO benchmark by adding Augmented Memory and BAR implementations. We further add experience replay to the implemented version of AHC , for comparison.…”

“…This is demonstrated in an exploitation task, the PMO benchmark, and in both drug and materials design case studies. We propose a method called Selective Memory Purge, which can be used in conjunction with Augmented Memory to generate diverse molecules while retaining enhanced sample efficiency. We explicitly highlight the importance of experience replay on the sample efficiency of REINVENT and all proposed algorithmic modifications. We expand the PMO benchmark by adding Augmented Memory and BAR implementations. We further add experience replay to the implemented version of AHC , for comparison. …”

Sample efficient reinforcement learning with active learning for molecular design

“…the number of oracle calls needed to reliably learn the desired output. REINVENT has been identified as one of the most sample-efficient generative chemical models; both in benchmarks which do not consider compound chemistry relative to the pre-training data 60 as well as benchmarks which do, 60 however the model still requires thousands of oracle evaluations to learn to produce favorable molecules. While this may compare favorably with the cost of brute-force VS on large libraries, the incorporation of higher-cost simulations remains prohibitive.…”

On The Difficulty of Validating Molecular Generative Models Realistically: A Case Study on Public and Proprietary Data

“…A real-world interpretation of generative models in the drug discovery context remains difficult, and the current work attempts to better understand this by retrospectively applying performance measures to generative models applied to public and private drug discovery data sources. The objective of the task is hence to achieve late-stage project compounds, given information from early-stage compounds, in a limited number of steps, and hence in a sample-efficient way (for a more detailed recent evaluation of the sample efficiency of different methods see a recent study 18 ). This early/late data split strategy is in analogy to 'time-split' validation in the QSAR area, where splitting data into training and test sets along the time domain has been proposed before 19 .…”