Deep inverse reinforcement learning for structural evolution of small molecules

Abstract

The size and quality of chemical libraries to the drug discovery pipeline are crucial for developing new drugs or repurposing existing drugs. Existing techniques such as combinatorial organic synthesis and high-throughput screening usually make the process extraordinarily tough and complicated since the search space of synthetically feasible drugs is exorbitantly huge. While reinforcement learning has been mostly exploited in the literature for generating novel compounds, the requirement of designing a reward function that succinctly represents the learning objective could prove daunting in certain complex domains. Generative adversarial network-based methods also mostly discard the discriminator after training and could be hard to train. In this study, we propose a framework for training a compound generator and learn a transferable reward function based on the entropy maximization inverse reinforcement learning (IRL) paradigm. We show from our experiments that the IRL route offers a rational alternative for generating chemical compounds in domains where reward function engineering may be less appealing or impossible while data exhibiting the desired objective is readily available.

ChemistGA: A Chemical Synthesizable Accessible Molecular Generation Algorithm for Real-World Drug Discovery

“…In drug discovery, experimental verification of molecular properties is very time-consuming and labor-intensive. Most DL generative models have not yet incorporated this challenge into the design process. − ,,− ,,− ,− To make ChemistGA more aware of this constraint by design, we propose an augmented ChemistGA, called R-ChemistGA, which simulates the real environment as further elucidated below.…”

“…In recent years, with the rapid development of deep learning (DL), many advanced models such as recurrent neural networks, − variational autoencoders (VAEs), − generative adversarial networks (GANs), − and reinforcement learning (RL) ,,,− have been adopted to design novel molecules with specific properties. At the same time, in order to properly evaluate and compare the performance of these generative models, several benchmarks , have also been developed.…”

A review of reinforcement learning in chemistry

“…The wide variety of reward functions used in the work described so far highlights the importance of reward function construction in drug design problems. To bypass this challenge, Agyemang et al 90 treated drug design as an inverse RL problem and inferred the reward function for the agent from the SMILES strings of known molecules with desirable properties. Molecule generation is handled by a multiple layer stack-augmented RNN, with PPO for policy learning.…”

Magicmol - A light-weighted pipeline for drug-like molecule evolution and quick chemical space exploration

“…In our work, we retrained a backbone network without using the same parameter or structure of former excellent works [43,7,31] and adopted the SELFIES presentation. Our model appears to have high performance throughout the whole training process with approximately 4.9 million parameters, only one-tenth of the backbone models of others [43].…”

“…In our work, we retrained a backbone network without using the same parameter or structure of former excellent works [43,7,31] and adopted the SELFIES presentation. Our model appears to have high performance throughout the whole training process with approximately 4.9 million parameters, only one-tenth of the backbone models of others [43]. Among our concepts, the sampling capacity matters [44], for the reason of efficiently searching for a proper evolution target, thus we take the model structure into consideration.…”