MolGPT: Molecular Generation Using a Transformer-Decoder Model

Bagal, Viraj; Aggarwal, Rishal; Vinod, P. K.; Priyakumar, U. Deva

doi:10.1021/acs.jcim.1c00600

Cited by 206 publications

(185 citation statements)

References 48 publications

Supporting

Mentioning

142

Contrasting

Order By: Relevance

“…De novo drug design (Hartenfeller and Schneider, 2010) through existing computer technology can speed up drug development and save research costs. The tasks involved in de novo drug design include molecular generation (Gómez-Bombarelli et al, 2016;Cao and Kipf, 2018;Jin et al, 2018;You et al, 2018;Madhawa et al, 2019;Popova et al, 2019;Zhang et al, 2019;Hong et al, 2020;Zang and Wang, 2020;Bagal et al, 2021), drug and drug interactions (DDI) (Li et al, 2021;Lin et al, 2021;Lyu et al, 2021;Zhao et al, 2021), disease associations (Ding et al, 2020;Lei and Zhang, 2020;Mudiyanselage et al, 2020;Lei X.-J. et al, 2021;Lei X. et al, 2021;Wang Y. et al, 2021;Lei and Zhang, 2021;Yang and Lei, 2021;Zhang et al, 2021), and so on.…”

Section: Introductionmentioning

confidence: 99%

Cross-Adversarial Learning for Molecular Generation in Drug Design

Cui

et al. 2022

Front. Pharmacol.

View full text Add to dashboard Cite

Molecular generation is an important but challenging task in drug design, as it requires optimization of chemical compound structures as well as many complex properties. Most of the existing methods use deep learning models to generate molecular representations. However, these methods are faced with the problems of generation validity and semantic information of labels. Considering these challenges, we propose a cross-adversarial learning method for molecular generation, CRAG for short, which integrates both the facticity of VAE-based methods and the diversity of GAN-based methods to further exploit the complex properties of Molecules. To be specific, an adversarially regularized encoder-decoder is used to transform molecules from simplified molecular input linear entry specification (SMILES) into discrete variables. Then, the discrete variables are trained to predict property and generate adversarial samples through projected gradient descent with corresponding labels. Our CRAG is trained using an adversarial pattern. Extensive experiments on two widely used benchmarks have demonstrated the effectiveness of our proposed method on a wide spectrum of metrics. We also utilize a novel metric named Novel/Sample to measure the overall generation effectiveness of models. Therefore, CRAG is promising for AI-based molecular design in various chemical applications.

show abstract

Section: Introductionmentioning

confidence: 99%

Cross-Adversarial Learning for Molecular Generation in Drug Design

Cui

et al. 2022

Front. Pharmacol.

View full text Add to dashboard Cite

show abstract

“…Deep language models have also been used for the generation of protein sequences [27,28] and molecules [29,30]. In [31], Kim et al combined a transformer encoder with a conditional variational autoencoder (cVAE) to achieve high performance molecule generation.…”

Section: Introductionmentioning

confidence: 99%

Crystal Transformer: Self-learning neural language model for Generative and Tinkering Design of Materials

Wei¹,

Li²,

Song³

et al. 2022

Preprint

View full text Add to dashboard Cite

Self-supervised neural language models have recently achieved unprecedented success, from natural language processing to learning the languages of biological sequences and organic molecules. These models have demonstrated superior performance in the generation, structure classification, and functional predictions for proteins and molecules with learned representations. However, most of the masking-based pre-trained language models are not designed for generative design, and their black-box nature makes it difficult to interpret their design logic. Here we propose BLMM Crystal Transformer, a neural network based probabilistic generative model for generative and tinkering design of inorganic materials. Our model is built on the blank filling language model for text generation and has demonstrated unique advantages in learning the "materials grammars" in terms of high-quality generation, interpretability, and data efficiency. It can generate chemically valid materials compositions with as high as 89.7% charge neutrality and 84.8% balanced electronegativity, which has more than 4 and 8 times higher enrichment compared to enhanced random sampling. The probabilistic generation steps allow it to recommend generation or tinkering actions with explanation, which captures known materials chemistry and makes it useful for materials doping. Our models can be trained with fewer than 40,000 materials formulas demonstrating their high data efficiency compared to other pre-trained protein or molecule models trained with millions of samples. We have applied our model to discover a set of new materials as validated using DFT calculations. Our work thus not only brings the unsupervised transformer language models based generative artificial intelligence to inorganic materials but also has the potential to guide the development of better generative design models in the domain of biology (proteins) and organic molecules. A user-friendly web app has been developed and can be accessed freely at www.materialsatlas.org/blmtinker.

show abstract

“…Many generative model techniques and architectures applied to de novo molecule generation exist. These models range from purely symbolic approaches such as genetic algorithms [1,2] to more recent machine learning (ML) approaches such as recurrent neural networks (RNNs) [3][4][5][6][7], transformers [8][9][10], variational autoencoders [11][12][13][14], generative adversarial networks [15][16][17], graph neural networks [18,19] and hybrid approaches that use ML to guide reinforcement learning (RL) in a heuristic action space [20]. These generative models can produce valid and novel molecules [21,22] and condition molecule generation towards a particular endpoint [21] (e.g., predicted bioactivity towards a protein target [4]) via optimization techniques such as, RL [4,20,23], Bayesian optimization [11,13], molecular swarm optimization [14] and Monte Carlo tree search [2,6].…”

Section: Introductionmentioning

confidence: 99%

Augmented Hill-Climb increases reinforcement learning efficiency for language-based de novo molecule generation

Thomas

O’Boyle

Bender

et al. 2022

Preprint

View full text Add to dashboard Cite

A plethora of AI-based techniques now exists to conduct de novo molecule generation that can devise molecules conditioned towards a particular endpoint in the context of drug design. One popular approach is using reinforcement learning to update a recurrent neural network or language-based de novo molecule generator. However, reinforcement learning can be inefficient, sometimes requiring up to 10^5 molecules to be sampled to optimize more complex objectives, which poses a limitation when using computationally expensive scoring functions like docking or computer-aided synthesis planning models. In this work, we propose a reinforcement learning strategy called Augmented Hill-Climb based on a simple, hypothesis-driven hybrid between REINVENT and Hill-Climb that improves sample-efficiency by addressing the limitations of both currently used strategies. We compare its ability to optimize several docking tasks with REINVENT and benchmark this strategy against other commonly used reinforcement learning strategies including REINFORCE, REINVENT (version 1 & 2), Hill-Climb and best agent reminder. We find that optimization ability is improved ~1.5-fold and sample-efficiency is improved ~45-fold compared to REINVENT while still delivering appealing chemistry as output. Diversity filters were used, and their parameters were tuned to overcome observed failure modes that take advantage of certain diversity filter configurations. Lastly, we find that Augmented Hill-Climb outperforms the other reinforcement learning strategies used on six tasks, especially in the early stages of training or for more difficult objectives. Overall, we hence show that AHC improves sample-efficiency for language-based de novo molecule generation conditioning via reinforcement learning, compared to the current state-of-the-art. This makes more computationally expensive scoring functions, such as docking, more accessible on a relevant timescale.

show abstract

MolGPT: Molecular Generation Using a Transformer-Decoder Model

Cited by 206 publications

References 48 publications

Cross-Adversarial Learning for Molecular Generation in Drug Design

Cross-Adversarial Learning for Molecular Generation in Drug Design

Crystal Transformer: Self-learning neural language model for Generative and Tinkering Design of Materials

Augmented Hill-Climb increases reinforcement learning efficiency for language-based de novo molecule generation

Contact Info

Product

Resources

About